EEC-490 Senior Design (CE)

1. EEC-490 Senior Design (CE) Kinect Programming Tutorial 1 Wenbing Zhao wenbing@ieee.org EEC484/584: Computer Networks

2. Kinect Application Development Requirement • Hardware: a relatively powerful Windows 7 PC/Laptop (Windows XP not supported) • Software • Visual Studio 2010 (or Express) for C# (install this first!) • Kinect for Windows SDK • For speech recognition • Speech platform runtime (32-bit & 64-bit) • Microsoft speech platform sdk • MS Kinect English language pack

3. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx • Create a new Windows Presentation Foundation app • File->New->Project… • Select “WPF Application”, name the project “KinectBasic”, click OK • Add reference to Kinect library • Right click References=>Add Reference… • Select Microsoft.Research.Kinect=>OK

4. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx • Define UI • TextBlock • Text • FontSize • Button • Content • Images • Leftmost image: change variable name to image • Middle image: change variable name to imageCmyk32 • Rightmost image: change variable name to depth

5. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx • Add events • When “Start” / “Stop” button is pushed • In Properties pane => Events tab • In “Click” field: can specify event method name • Otherwise, a default method name is used (in the form buttonName_Click())

6. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx • MainWindow.xaml.cs • Add header include • using Microsoft.Research.Kinect.Nui; • Add member variable • In public partial class MainWindow:Window { • Runtime kinectNui; • int totalFrames = 0; // for frame rate calculation • int lastFrames = 0; // frame rate • DateTime lastTime = DateTime.MaxValue; // frame rate • const int RED_IDX = 2; • const int GREEN_IDX = 1; • const int BLUE_IDX = 0; • byte[] depthFrame32 = new byte[320 * 240 * 4]; // for frame manipulation

7. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx • MainWindow.xaml.cs • Add Kinect init method: private void InitializeNui() {} • Declares kinectNui as a Runtime object, which represents the Kinect sensor instance • nui = Runtime.Kinects[0]; • Open the video and depth streams, and sets up the event handlers that the runtime calls when a video, depth, or skeleton frame is ready • An application must initialize the Kinect sensor by calling Runtime.Initialize before calling any other methods on the Runtime object • kinectNui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

8. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx • MainWindow.xaml.cs, InitializeNui{} • To stream color images: • The options must include UseColor • Valid image resolutions are Resolution1280x1024 and Resolution640x480 • Valid image types are Color, ColorYUV, and ColorYUVRaw • kinectNui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.ColorYuv);

9. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs, InitializeNui{} To stream depth and player index data: The options must include UseDepthAndPlayerIndex Valid resolutions for depth and player index data are Resolution320x240 and Resolution80x60 The only valid image type is DepthAndPlayerIndex kinectNui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);

10. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs, InitializeNui{} Init timestamp for frame rate calculation: lastTime = DateTime.Now; Add event handler to handle incoming frames kinectNui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiVideoFrameReady); kinectNui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiDepthFrameReady);

11. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs Video frame ready event handler void NuiVideoFrameReady(object sender, ImageFrameReadyEventArgs e) { PlanarImage Image = e.ImageFrame.Image; image.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, Image.Bits, Image.Width * Image.BytesPerPixel); imageCmyk32.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Cmyk32, null, Image.Bits, Image.Width * Image.BytesPerPixel); }

12. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: Depth frame ready event handler Depth is different because the image you are getting back is 16-bit and we need to convert it to 32-bit void NuiDepthFrameReady(object sender, ImageFrameReadyEventArgs e) { var Image = e.ImageFrame.Image; var convertedDepthFrame = convertDepthFrame(Image.Bits); depth.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, convertedDepthFrame, Image.Width * 4); CalculateFps(); }

13. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: convert depth frame Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame that displays different players in different colors byte[] convertDepthFrame(byte[] depthFrame16) { for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4){ int player = depthFrame16[i16] & 0x07; int realDepth = (depthFrame16[i16 + 1] << 5) | (depthFrame16[i16] >> 3); // transform 13-bit depth information into an 8-bit intensity appropriate // for display (we disregard information in most significant bit) byte intensity = (byte)(255 - (255 * realDepth / 0x0fff)); depthFrame32[i32 + RED_IDX] = intensity; depthFrame32[i32 + BLUE_IDX] = intensity; depthFrame32[i32 + GREEN_IDX] = intensity; } return depthFrame32;}

14. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: calculate frame rate void CalculateFps() { ++totalFrames; var cur = DateTime.Now; if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1)) { int frameDiff = totalFrames - lastFrames; lastFrames = totalFrames; lastTime = cur; frameRate.Text = frameDiff.ToString() + " fps"; } }

15. Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: windows loading/closing private void WindowLoaded(object sender, RoutedEventArgs e) { InitializeNui(); } private void WindowClosed(object sender, EventArgs e) { kinectNui.Uninitialize(); } • Time to run the app!