1 / 34

Kinect Interactions: Get a Grip in C# or HTML5/ Javascript

Kinect Interactions: Get a Grip in C# or HTML5/ Javascript. Ben Lower Program Manager – Kinect for Windows 3-705. Agenda. Periodic Table InteractionStream Sneak Peek of 1.8 SDK New Kinect? Q&A. Periodic Table. CHUCK NORRIS HAD TO DESTROY THE PERIODIC TABLE….

metta
Download Presentation

Kinect Interactions: Get a Grip in C# or HTML5/ Javascript

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Kinect Interactions: Get a Grip in C# or HTML5/Javascript Ben Lower Program Manager – Kinect for Windows 3-705

  2. Agenda • Periodic Table • InteractionStream • Sneak Peek of 1.8 SDK • New Kinect? • Q&A

  3. Periodic Table

  4. CHUCK NORRIS HAD TO DESTROY THE PERIODIC TABLE… HE ONLY RECOGNIZES THE ELEMENT OF SURPRISE!

  5. The Periodic Table of Kinect for Windows A Stream Symbol Audio Stream Name Data Type Waveform

  6. Data Streams (Eyes & Ears) IR D A C AC Depth Color Accelerometer Infrared Audio DepthImagePixel[] Waveform byte[] byte[] Vector4

  7. Recognition Streams (Brains) I KF FT S ST Interactions Kinect Fusion Face Tracking Speech Skeleton Tracking InteractionFrame (UserInfo[]) Point Cloud/Mesh FaceTrackFrame Recognition Events Skeleton[]

  8. Demo: Controls Basics + Interaction Gallery

  9. InteractionStream

  10. Why InteractionStream? • Moving to world that is multi-modal • Have experiences feel natural • Consistency of interactions

  11. Interaction Stream • Translates hand position to screen coordinates • Physical Interaction Zone (PhIZ) is shaped to be ergonomic for an individual independent of position relative to the sensor • Track and recognize press action • Press action based on arm extension • Press distance dependent on starting arm position • Press adjusted to improve accuracy over buttons • Recognize hand grip and release gesture • Computer vision and machine learning algorithm recognize grip gesture • Grip recognized for a wide variety of hand poses and orientations

  12. The PhIZ • Physical Interaction Zone (PhIZ) • The area in front of the user where there is a spatial mapping relationship between the user’s hand movements in physical space and the cursor movement on the screen. • Features • Ergonomics and Human Movement • Scaling to fit user • Separate PhIZ for each hand • Consistent experience

  13. Targeting & Selection Old Way Target button with hand cursor Hover, hold, wait (for action) New Way Target button with hand cursor Press toward button, then release Center “pulls hand in” during press Cursor Default (L) Cursor Default (R)

  14. Press Cursors Research showed people like visual cues & feedback Hand “fills up” to indicate press progress Full Press Press Complete Roll Over Press Half

  15. Scrolling & Direct Manipulation Old Way Target scroll button w/hand cursor Hold and wait for scroll New Way Target anywhere w/hand cursor Grip to manipulate Cursor Default Grip Over

  16. InteractionStream APIs • Native COM based InteractionStream API • Core implementation • Usable with any UI frameworks that can call C functions and COM objects • Managed .NET Wrapper • Wraps the InteractionStream API for .NET languages, e.g. C#, VB etc. • WPF Adapter – KinectRegion, Cursor and Controls • Implements IInteractionClient for hit-testing WPF controls • Translates Interaction Frame to WPF events • Implements Cursor to provide feedback for: handedness, press progress and grip state

  17. Interactions Architecture C++/DirectX Kinect-Enabled App C++/Win32 Kinect-Enabled App XNA Kinect-Enabled App Windows Forms Kinect-Enabled App .NET/WPF Kinect-Enabled App WPF Kinect Controls Microsoft.Kinect.Toolkit.Controls.dll C++/Win32 Kinect Controls DirectX Kinect Support XNA Kinect Support WindowsForms Kinect Controls InteractionStream Microsoft.Kinect.Toolkit.Interaction.dll NUI_INTERACTION STREAM KinectInteraction170_*.dll (32 or 64)

  18. Working with InteractionStream

  19. Integration Considerations • Primary User and Primary Hand • InteractionStream tracks 2 users and 4 hands • Primary User and Primary Hand implemented by KinectRegion • UX Guidance: Avoid 2 handed actions – difficult to learn & master • Cursor – press and grip visualization • Continuous press progress feedback critical to user experience • Press initiates action on release – similar to button press with mouse • Pressable button affordance • Button provides visual feedback when cursor is over pressable button

  20. Sneak Peek: v1.8 SDK

  21. Demo: Background Removal

  22. Demo: Photobombr

  23. Interactions Architecture (1.8) C++/DirectX Kinect-Enabled App C++/Win32 Kinect-Enabled App XNA Kinect-Enabled App Windows Forms Kinect-Enabled App .NET/WPF Kinect-Enabled App HTML5/Javascript Kinect-Enabled App HTML5/JS Controls Microsoft.Samples. Kinect.Webserver.dll WPF Kinect Controls Microsoft.Kinect. Toolkit.Controls.dll C++/Win32 Kinect Controls DirectX Kinect Support XNA Kinect Support WindowsForms Kinect Controls InteractionStream Microsoft.Kinect.Toolkit.Interaction.dll NUI_INTERACTION STREAM KinectInteraction170_*.dll (32 or 64)

  24. Announcing: v1.8 SDK (September 2013) • Background Removal API (green screening w/out the screen) • HTML5/Javascript support for Interactions • Kinect Fusion Color • Plus more…stay tuned

  25. What’s up with the new Kinect?

  26. New Kinect for Windows Sensor • Wider field of view • HD resolution • Track six skeletons • Better ST, audio, infrared • Coming next year

  27. Announcing: Developer Kit Program • Direct access to our engineering team • Exclusive access to pre-release SDKs • Private access to all API & sample docs • Pre-release/alpha sensor • New RTM sensor next year $399 (coming November 2013) Apply now: http://aka.ms/k4wdevkitapplication

  28. Who should apply? • Already working today to build & deploy NUI experiences • You are interested in or experimenting with NUI • You want to help push the boundaries of NUI • Experienced developer who wants to jump in • You have an awesome idea for a touch-free app

  29. Thank you! • Get 1.7 SDK now • Follow our blog to know when 1.8 SDK is released • Apply today for new dev kit program

  30. Kinect for Windows @ Build 2013 • All Week • Kinect for Windows Booth (demos, 3D printing with Fusion) • Wednesday, June 26th • 2:00 PM Real-world Machine Learning: How Kinect Gesture Recognition Works, 3-704 Alisson Sol, Bing (formerly Kinect for Windows), North Hall: 123 • Thursday, June 27th • 2:00 PM Kinect Interactions: Get a Grip in C# or HTML5/Javascript, 3-705 Ben Lower, Kinect for Windows, North Hall: 131 • 5:00 PM Scanning & Printing the Real-World using Kinect for Windows and Preview of vNext, 3-702 Chris White, Kinect for Windows, North Hall: 131

  31. Kinect for Windows Resources • Developer Kit Program Application • http://aka.ms/k4wdevkitapplication • Developer Center (SDKs, Samples, Videos) • http://aka.ms/k4wdev • Product Info • http://aka.ms/k4w

More Related