1 / 38

Gesture Interfaces

Gesture Interfaces. Davide Spano lucio.davide.spano@isti.cnr.it spano@di.unipi.it. CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 1 56124 Pisa, Italy. Gestures. Remotes (1). Nintendo Wiimote , 2006 Buttons Three-axis linear accelerometer IR camera

iokina
Download Presentation

Gesture Interfaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Gesture Interfaces Davide Spanolucio.davide.spano@isti.cnr.it spano@di.unipi.it CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 156124 Pisa, Italy

  2. Gestures Davide Spano Gesture Interfaces

  3. Remotes (1) • Nintendo Wiimote, 2006 • Buttons • Three-axis linear accelerometer • IR camera • Three axis gyroscope extension (Wii Motion Plus) • Game market opened to casual gamers Davide Spano Gesture Interfaces

  4. Remotes (2) • Playstation Move, 2010 • Buttons • Spherical RGB light emitter plus camera • Three-axis accelerometer and gyroscope • Compass Davide Spano Gesture Interfaces

  5. Floor Devices (1) • Dance pads • Konami Dance Dance Revolution (1998) • Huge directional pads • Buttons Davide Spano Gesture Interfaces

  6. Floor Devices (2) • Wii Balance Board • Four pressure sensors • Snowboard games • Fitness games Davide Spano Gesture Interfaces

  7. Motion Tracking (1) • CamSpace, 2009 • Object recognition • No haptic feedback Davide Spano Gesture Interfaces

  8. Motion Tracking (2) • Microsoft Kinect, 2009 • RGB camera • Depth sensor • Microphone array • Still no haptic feedback Davide Spano Gesture Interfaces

  9. Multitouch • Two or more touch-sensed position on the screen • Patents already on 1984 • iPhone 2007 • Support in Mac OS X, Windows 7, Ubuntu … Davide Spano Gesture Interfaces

  10. Gestures Overview • Gesture types: • Static • Dynamic • Spatio-temporal • Conceptual: • Gestures are continuous • Gestures exploit time and space as arguments • Gestures enable single user parallel interaction • Need for a suitable programming paradigm Davide Spano Gesture Interfaces

  11. The Midas Touch Out Of range Tracking Engaged Mouse Touches, Skeleton Accelerometers, Gyroscope Not Tracking Tracking Tracking Davide Spano Gesture Interfaces

  12. Current Programming Model • Do gestures really fit this pattern? Davide Spano Gesture Interfaces

  13. Multitouch Frameworks Davide Spano Gesture Interfaces

  14. Example: Pinch for zoom (1) Davide Spano Gesture Interfaces

  15. Example: Pinch for zoom (2) Start Move Move End Davide Spano Gesture Interfaces

  16. Full body tracking Davide Spano Gesture Interfaces

  17. Microsoft Kinect SDK Davide Spano Gesture Interfaces

  18. Primesense NITE Controls • State of the Art toolkit for full body gesture interaction • Reusable controls (recognizers) for NUI • Control list • Push detector • Swipe detector • Steady detector • Wave detector • Circle detector • Selectable slider (1D/2D) • All based on the observer pattern Davide Spano Gesture Interfaces

  19. Circle Control Sample Control Event Skeleton Tracking Davide Spano Gesture Interfaces

  20. One Event is not enough! Handler Gesture Handler Handler 1 1 1 Pan Handler Touch ID Samples Start End Move Davide Spano Gesture Interfaces

  21. Gesture Definition Meta Model • Declarative • Compositional • Ground Terms (basic building blocks) • Composition Operators • Behaviour Association • Entire Gesture • Any other sub-component • Abstract Gesture Recognition Support • Formally defined through Petri Nets Davide Spano Gesture Interfaces

  22. Break… http://www.foxtrot.com/ Davide Spano Gesture Interfaces

  23. Gesture Support Abstraction • Feature: an observable property • Gesture Support: a set of features • Gesture Support State: feature values at a given time • Gesture Support Sequence: gesture support state values through the time Davide Spano Gesture Interfaces

  24. Ground Terms and Composition Operators • Low-level device events as ground terms • Optional conditions on support state • Composition operators Davide Spano Gesture Interfaces

  25. Multitouch Gesture Support • Features: • 2D position of n touches, in pixel coordinates • If a touch is not currently detected • current detection time • Multi-Touch recognition ground terms: Davide Spano Gesture Interfaces

  26. Multitouch Gesture Description • Tap • Double Tap • Pan • Slide • Pinch • Rotate Davide Spano Gesture Interfaces

  27. Proton (1) • Kin et al. Proton: Multitouch Gestures as Regular Expressions, CHI 2012 • Multitouch Gestures Described as Regular Expression • Ground Terms • Composition operators • * • Matcher for handling ambiguity • One gesture at time assumption • Tablature graphical representation for gestures Davide Spano Gesture Interfaces

  28. Proton (2) Davide Spano Gesture Interfaces

  29. Full Body Gesture Platform • Skeleton tracked as a set of joints • 2 user full tracking + 4 only body centre • Features: • Joint (user id, joint id, position, orientation) • Current time Davide Spano Gesture Interfaces

  30. Full Body Example (1) Front Not Front Right hand Both hands Davide Spano Gesture Interfaces

  31. Full Body Example Davide Spano Gesture Interfaces

  32. A different programming model • Describe declaratively the gesture • Create new ones using the building blocks • Reuse existing ones through composition • Code the event handlers • Attach them to the events at the desired granularity • It is called only when it is needed (no further checks) • Temporal description and interface behaviour are completely separated • Complex Gestures can be stored as classes • They can still expose their low level events Davide Spano Gesture Interfaces

  33. Library Support Davide Spano Gesture Interfaces

  34. Multitouch with Full Body Experiment • Hands point tracking • Basic Touches Description Davide Spano Gesture Interfaces

  35. Proof of Concept Applications Davide Spano Gesture Interfaces

  36. Demo Videos • Parallel Interaction with iPhone video • “Multitouch” with Kinect video • Same Gesture Definition! Davide Spano Gesture Interfaces

  37. Conclusion http://www.mojocomic.com/ Davide Spano Gesture Interfaces

  38. References • Spano, L.D. A Model Based Approach for gesture interfaces (EICS 2011) pdf • Kin K., Hartmann B., DeRoseT., Agrawala M. Proton: Multitouch Gestures as Regular Expressions pdf • Kinect for Windows • OpenNI • Primesense NITE Davide Spano Gesture Interfaces

More Related