1 / 23

EEC-492/592 Kinect Application Development

Learn about the depth data stream in Kinect application development, including how depth measurement is done, capturing and processing depth data, and classes related to depth data.

Download Presentation

EEC-492/592 Kinect Application Development

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. EEC-492/592Kinect Application Development Lecture 7 Wenbing Zhao wenbing@ieee.org

  2. Outline Depth data stream How depth measurement is done Capture and process depth data Classes related to depth data Getting the Distance from a Particular Pixel

  3. Depth Data Stream • Depth data stream consists of a sequence of depth image frames • Each depth image frame is in the form of a 16-bit grey scale image • Viewable range: 43 degrees vertical, 57 degrees horizontal • A depth pixel contains the following • (x,y) coordinates represent the location of the object in the depth sensor view • The value of the pixel at (x,y) contains the distance between the object and the Kinect sensor in millimeters

  4. Depth Data Stream • Depth data stream supports • 640x480 at 30fps • 320x240 at 30fps • 80x60 at 30fps • Kinect depth sensing range • Up to 13.1 feet (4 meters) • As close as 40cm

  5. How the Depth Data is Obtained

  6. Stereo Triangulation • Stereo triangulation is an analysis algorithm for computing the 3D position of points in an image frame • In general stereo triangulation, two images are used to obtain the two different views on a scene, in a similar fashion to human binocular vision. • By comparing these two images, the relative depth information is calculated • Kinect stereo triangulation uses one real (IR) image and a virtual image (more accurately, the line of each projected point) • http://www.youtube.com/watch?v=uq9SEJxZiUg

  7. Depth Measurement • When there is a pixel in any of the diagonal views of the sensor, it internally draws a line that is perpendicular to the sensor and then calculates the distance directly from there

  8. Capturing and Processing Depth Data • Enable the depth stream channel with the type of depth image format • Attach the event handler to the stream channel • Process the incoming depth frames • Render the frames on UI this.sensor = KinectSensor.KinectSensors[0]; this.sensor.DepthStream.Enable(); sensor.DepthFrameReady += depthFrameReady); void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e) { }

  9. Processing Depth Data void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e) { using (DepthImageFrame imageFrame = e.OpenDepthImageFrame()) { if (null == imageFrame) { return; } imageFrame.CopyPixelDataTo(depthPixels); // Write the pixel data into our bitmap this.depthBitmap.WritePixels( new Int32Rect(0, 0, this.depthBitmap.PixelWidth, this.depthBitmap.PixelHeight), this.depthPixels, stride, 0); } } Very similar to the code for ColorImage Processing: color => depth

  10. Classes Related to Depth Data

  11. Classes Related to Depth Data

  12. Depth Data and Distance • The DepthImageStream and DepthImageFrame classes have the MaxDepth and MinDepth properties, which return the maximum and minimum depth ranges for that particular stream or captured image frame, in millimeters • This range value returns the best range where distance can be measured properly • These values will change automatically based on the selection of DepthRange for the stream

  13. Depth Range • The Range property sets the viewable range for the Kinect sensor, which is a type of DepthRange enumeration • A DepthRange enumeration has the following two values: • Default • Near sensor.DepthStream.Range = DepthRange.Near;

  14. Special Depth Range Values • DepthImageStream defines three additional read-only properties • TooFarDepth: It provides the depth point where the object is beyond the range of the sensor • TooNearDepth: It provides the depth point where the object is too close to the sensor • UnknownDepth: There are instances when a depth is completely indeterminate, and the value will be zero

  15. Default Depth Range

  16. Depth Data Representation • The Kinect sensor returns 16-bit raw depth frame data • The first three bits are used to represent the identified players and the remaining 13 bits give you the measured distance in millimeters • To retrieve the 13-bit depth data, perform a bitwise shift operation >> to move the bits to the beginning int depth = depthFrame[depthIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth;

  17. Depth Data Representation

  18. Getting the Distance from a Particular Pixel • Add feature to our DepthCam app • Very important: make the image control exactly the same size as the resolution chosen • Image should be 320x240 if Resolution320x240Fps30 • Image should be 640x480 if Resolution640x480Fps30 • Use mouse to select a pixel in the image => display the depth for that pixel • Need to add event handler for MouseDown Point currentPoint = e.GetPosition(depthImageControl); int pixelIndex = (int)(currentPoint.X + ((int)currentPoint.Y * this.frame.Width)); int distancemm = this.pixelData[pixelIndex] >> DepthImageFrame. PlayerIndexBitmaskWidth;

  19. Build the DepthCam App • Create a new WFP project named DepthCam • Draw the GUI for the app • Modify xaml file to add WindowLoaded and WindowClosing as before • Add Microsoft.Kinect reference • Add member variables for displaying depth images • Start Kinect sensor, enable depth data stream, register depth stream handler, connect image control to the bitmap for display • Process each depth image • Add code for displaying depth data for pixel clicked

  20. GUI Design • Image control • Group control for depth information • Two sets of labels and two sets of textboxes • Use meaningful names for the textboxes • Group control for pixel information • Four sets of labels and four sets of textboxes

  21. Adding Code KinectSensor sensor; private WriteableBitmap depthBitmap; private short[] depthPixels; private int frameWidth; // to calculate depthIndex • Add member variables: • WindowLoade method (WindowClosing() same as before): private void WindowLoaded(object sender, RoutedEventArgs e) { if (KinectSensor.KinectSensors.Count > 0) { this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) { this.sensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30); this.depthPixels = new short[this.sensor.DepthStream.FramePixelDataLength]; this.depthBitmap = new WriteableBitmap(this.sensor.DepthStream.FrameWidth, this.sensor.DepthStream.FrameHeight, 96.0, 96.0, PixelFormats.Gray16, null); this.image1.Source = this.depthBitmap; this.sensor.DepthFrameReady += this.depthFrameReady; this.sensor.Start(); } else { MessageBox.Show("No device is connected!"); this.Close(); } } }

  22. Adding Code void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e) { using (DepthImageFrame imageFrame = e.OpenDepthImageFrame()) { if (null == imageFrame) { return; } this.frameWidth = imageFrame.Width; this.maxDepthField.Text = "" + imageFrame.MaxDepth; this.minDepthField.Text = "" + imageFrame.MinDepth; imageFrame.CopyPixelDataTo(depthPixels); int stride = imageFrame.Width * imageFrame.BytesPerPixel; this.depthBitmap.WritePixels( new Int32Rect(0, 0, this.depthBitmap.PixelWidth, this.depthBitmap.PixelHeight), this.depthPixels, stride, 0); } } • Event handler for depth frames:

  23. Adding Code • Add mouse down event on image control • Find the MouseDown line, double click it to get event handler template • Event handler for mouse click: private void image1_MouseDown(object sender, MouseButtonEventArgs e) { Point currentPoint = e.GetPosition(image1); this.pixelXField.Text = currentPoint.X.ToString(); this.pixelYField.Text = currentPoint.Y.ToString(); int pixelIndex = (int)(currentPoint.X + ((int)currentPoint.Y * this.frameWidth)); this.depthIndexField.Text = "" + pixelIndex; int distancemm = this.depthPixels[pixelIndex] >> depthImageFrame.PlayerIndexBitmaskWidth; this.depthField.Text = "" + distancemm; }

More Related