Windows Touch Deep Dive

1. CL13 Windows Touch Deep Dive Reed Townsend Program Manager Microsoft Corporation

2. Agenda • Introduction • Overview of Windows Touch API surface • Real-world ISV questions and solutions • Demo projects • 3D manipulations • Reimplementing Collage’s PhotoStrip control • Wrap up

3. Touch - Framing the Opportunity • What is touch? • It’s an input method • What does it mean to design for touch? • When designing for inputs, we’re really talking about user interface design (in the broad sense) • UI is often synonymous with visuals and the input and interaction half is forgotten

4. Touch - Framing the Opportunity • 99% of design is done in the platform • Controls and APIs are the language of UI • “The limits of my language mean the limits of my world” – Ludwig Wittgenstein • Games (and sometimes web) follow a different model • What are the common controls for touch? ?

5. Windows Touch APIs • Good/Better/Best platform model • Good: • What you get for free • Legacy, non-touch-aware apps • Better • Minimal investment • Typically WM_GESTURE, mouse message inspection • Best • Enables touch-optimized experiences! • WM_TOUCH, manipulations & inertia, and more

6. WM_GESTURE • Provides notifications when the user performs gestures over your window • Contains additional information like center of gesture and gesture-specific arguments Typical sequence: WM_TABLET_QUERYSYSTEMGESTURESTATUS WM_GESTURENOTIFY WM_GESTURE (GID_BEGIN) WM_GESTURE (GID_PAN, GF_BEGIN) WM_GESTURE (GID_PAN, no flags) ... WM_GESTURE (GID_PAN, GF_INERTIA) ... WM_GESTURE (GID_PAN, GF_END) WM_GESTURE (GID_END)

7. Real-world Questions and Answers WM_GESTURE Q: I registered for WM_TOUCH but I no longer get WM_GESTURE. What gives? A: Yes, WM_GESTURE and WM_TOUCH are exclusive in Win7. Q: I’m implementing gestures in my app, but I stopped getting panning bounce feedback. Why? A: Two possibilities: Panning messages are being accidentally eaten, or your app is providing custom handling of them but not calling the bounce feedback API.

8. GetSystemMetrics() • GetSystemMetrics(): • SM_DIGITIZER returns data about available digitizers on the system • SM_TABLETPC returns data about Tablet functionality • SM_MAXIMUMTOUCHES yields the largest number of contacts any of the available digitizers supports

9. Mouse Messages & GetMessageExtraInfo() • GetMessageExtraInfo() returns the extra info associated with a message • Mouse up and down messages are tagged with a special signature indicating they came from touch or pen: • Mask extra info against 0xFFFFFF80 • 0xFF515780 for touch, 0xFF515700 for pen

10. Richer Touch Experiences demo WM_TOUCH Applications

11. WM_TOUCH • Provides raw touch input data, conceptually similar to mouse messages • Finger painting, custom gestures, feeding higher-level controls (e.g. Manipulations) Typical sequence: WM_TOUCH (first finger down) WM_TOUCH (first finger moving) ... WM_TOUCH (two fingers moving, third comes down) WM_TOUCH (three fingers moving) ... WM_TOUCH (one goes up, two fingers moving) WM_TOUCH (two fingers moving) ... WM_TOUCH (last finger goes up)

12. Real-world Questions and Answers WM_TOUCH Q: How can I disable the press and hold gesture in my window? A: WM_TABLET_QUERYSYSTEMGESTURESTATUS can be used to disable P&H, flicks, and tap feedback. (WM_TOUCH windows already disable flicks by default. SetGestureConfig() controls gestures available via WM_GESTURE.) Q: I’m building a UI framework. How can I add touch support? A: Start with WM_TOUCH. If you want pan/zoom/rotate, use manipulations on top of WM_TOUCH.

13. Touch-Optimized Experiences demo WM_TOUCH, Manipulations, and More

14. Manipulations & Inertia • Manipulations: • Provides the full range of 2D affine transforms corresponding to user input • Enables multiple concurrent interactions • Can be fed by any source of raw data • Are a superset of WM_GESTURE support • E.g. Rotating and translating 3 photos simultaneously

15. Typical Flow On contact down: • Hit test against targets • (Add to map of contact ids and MPs) • MP->ProcessDown() On contact moves: • (Look up mapping) • MP->ProcessMove() On contact up: • (Look up mapping) • MP->ProcessUp() • (Kick off inertia) Get callbacks on event interface • Apply transforms to targets

16. Advanced Manipulation Topics • Single-finger rotation • PivotPoint, PivotRadius properties • Rotation around a point • Generalization of single-finger rotation • Supporting mouse-based manipulations • All covered in the Platform SDK manipulations sample

17. Inertia • Grab initial state from manipulation processor in completed event • Set up deceleration properties • Set up borders/margins • Call Process() on a timer

18. Demo Projects! • Demos: • 3D manipulations – An approach to mapping 2D onto 3D. • Reimplementing Collage’s PhotoStrip– Building custom touch controls • Examples of new ways to use the platform and new user interface options

19. 3D Manipulations • Set up 3D view, axes, objects • Add WM_TOUCH • Add manipulations • Map manipulations to 3D navigation • 2 finger camera operations, zoom • Spherical panning • Add inertia and other tweaks

20. 3D Manipulations • Classes of note: • D3DXDriver • CComTouchDriver • Camera • Inherits from InertiaObj and _IManipulationEvents • Example data flow: • WM_TOUCH message arrives • CComTouchDriver::ProcessInputEvent() • Camera::ProcessInputEvent() • InertiaObj::ProcessInputEvent() • Manipulation Processor • Camera::ManipulationDelta() • Update the camera

21. Building a PhotoStrip Control • Part of a control framework • Respond to touch messages • Add constrained manipulations for panning • Add support for dragging in/out of control • Add API surface • Flesh out overlay and gallery

22. Building a PhotoStrip Control • Window hierarchy: • Example data flow: • WM_TOUCH arrives • Hits overlay, routed to appropriate control • Photostrip gets message • Touch pans and triggers photo removal • Message to app window • Message to gallery • Photo is displayed in gallery Overlay HWND PhotoStrip1 HWND Gallery HWND PhotoStrip2 HWND

23. Implementation Notes • Smoothing input and manipulation data • Mixed input layers • 3D: manipulations + gestures • PhotoStrip: raw data + manipulations • Supporting multiple interactions across windows • Overlay + forwarding approach • Capture • Mediating between inputs

24. Final Thoughts • What are the common controls for touch? • We don’t know – they need to be defined • The existing controls were optimized for mouse and keyboard over decades • Help shape the future of touch user interfaces!

25. Touch Development Roadmap .NET 4.0 & Surface Toolkit (Q1 2010) Native Win32 Application WinFormsApplication WPF Application Surface Application Surface SDK 1.0 Surface Multi-Touch Controls &API Multi-Touch Controls & API Multi-Touch Controls Surface Toolkit for Windows 7 Managed Wrapper and Interop WPF 3.5 SP1 WPF 4.0 w/ Multi Touch APIs Surface Hardware Windows Vista Windows 7 Multi-Touch API

26. Related Content and Resources Workshops and sessions: • PDC09-WKSP08 – Windows 7 Developer Boot Camp • PDC09-CL27 – Multitouch on Microsoft Surface and Windows 7 for .NET Developers Contact us: • Forums: http://social.msdn.microsoft.com/Forums/en-US/tabletandtouch/threads/ • Email: wintouch@microsoft.com

28. Extra Slides

29. WM_GESTURE • Provides notifications when the user performs gestures over your window • Contains additional information like center of gesture and gesture-specific arguments Typical sequence: WM_TABLET_QUERYSYSTEMGESTURESTATUS WM_GESTURENOTIFY WM_GESTURE (GID_BEGIN) WM_GESTURE (GID_PAN, GF_BEGIN) WM_GESTURE (GID_PAN, no flags) ... WM_GESTURE (GID_PAN, GF_INERTIA) ... WM_GESTURE (GID_PAN, GF_END) WM_GESTURE (GID_END)

30. Controlling Gesture Behavior • WM_GESTURENOTIFY – “Heads up, a gesture is coming” • SetGestureConfig() • Control how gestures work in your window • Examples: 2D panning, explicitly setting SFP flags • Can set in response to WM_GESTURENOTIFY, at window creation, or other times

31. Single-Finger Panning (SFP)Enabling 2D panning GESTURECONFIG gc = { GID_PAN, GC_PAN & GC_PAN_WITH_SINGLE_FINGER_VERTICALLY & GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY & GC_PAN_WITH_INERTIA, 0 }; BOOL bResult = SetGestureConfig( hWnd, // window for which configuration is specified 0, // reserved, must be 0 1, // count of GESTURECONFIG structures &gc, // array of GESTURECONFIG structures sizeof(GESTURECONFIG) // sizeof(GESTURECONFIG) );

32. Legacy Gesture Handling • If an app doesn’t handle WM_GESTURE, it is treated as a gesture-unaware legacy app • Panning and zooming fallback behaviors are provided • If you want any default handling of gestures, make sure GID_BEGIN, GID_END, and the GID_ for the gesture in question are passed to DefWindowProc()

33. Real-world Questions and Answers WM_GESTURE Q: I registered for WM_TOUCH but I no longer get WM_GESTURE. What gives? A: Yes, WM_GESTURE and WM_TOUCH are exclusive in Win7. Q: I’m implementing gestures in my app, but I stopped getting panning bounce feedback. Why? A: Two possibilities: Panning messages are being accidentally eaten, or your app is providing custom handling of them but not calling the bounce feedback API.

34. GetSystemMetrics() • GetSystemMetrics(): • SM_DIGITIZER returns data about available digitizers on the system • SM_TABLETPC returns data about Tablet functionality • SM_MAXIMUMTOUCHES yields the largest number of contacts any of the available digitizers supports

35. Mouse Messages & GetMessageExtraInfo() • GetMessageExtraInfo() returns the extra info associated with a message • Mouse up and down messages are tagged with a special signature indicating they came from touch or pen: • Mask extra info against 0xFFFFFF80 • 0xFF515780 for touch, 0xFF515700 for pen

37. Advanced WM_GESTURE Topics • Compound gestures: • Zoom and Pan simultaneously: both messages contain information about both gestures • Message bubbling • Unhandled gestures bubble up the parent window chain • If they aren’t handled, OS legacy support kicks in • Coalescing • WM_GESTURE message are coalesced • Capture • Similar to mouse, implicit capture to window on down

38. Advanced WM_TOUCH Topics • Primary contact • First finger down is the “primary contact” and determines the target window for capture • Capture • Similar to mouse, contacts are implicitly captured on down and limited to the primary contact’s window • Coalescing • WM_TOUCH is coalesced. Can disable coalescing explicitly w/ a flag in RegisterTouchWindow().

39. Touch Development Roadmap Windows 7 (2009) Native Win32 Application WinFormsApplication WPF Application Surface Application Surface SDK 1.0 Multi-Touch Controls Multi-Touch Controls & API WPF 3.5 SP1 Managed Wrapper and Interop Surface Hardware Windows Vista Windows 7 Multi-Touch API

40. Touch Development Roadmap .NET 4.0 & Surface Toolkit (Q1 2010) Native Win32 Application WinFormsApplication WPF Application Surface Application Surface SDK 1.0 Surface Multi-Touch Controls &API Multi-Touch Controls & API Multi-Touch Controls Surface Toolkit for Windows 7 Managed Wrapper and Interop WPF 3.5 SP1 WPF 4.0 w/ Multi Touch APIs Surface Hardware Windows Vista Windows 7 Multi-Touch API

41. Touch Development Roadmap Surface vNext (Future) Native Win32 Application WinFormsApplication WPF Application Surface Application Surface SDK vNext Surface Multi-Touch Controls &API Multi-Touch Controls Surface-specific Controls & API Surface Toolkit Managed Wrapper and Interop WPF 4.0 w/ Multi Touch APIs Surface Hardware Windows 7 Windows 7 Multi-Touch API

42. YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation forms online at MicrosoftPDC.com

43. Learn More On Channel 9 • Expand your PDC experience through Channel 9 • Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses channel9.msdn.com/learn Built by Developers for Developers….