Developing For Microsoft Surface

1. PC17 Developing ForMicrosoft Surface  Brad Carpenter General Manager Surface Platform

2. Agenda • Microsoft Surface and the future of computing • How youcan be part of this revolution • How the Surface SDK enables rapid innovation • Microsoft’s multi-touch roadmap • Questions and Answer

3. Interface Paradigm Shift CLI GUI NUI

4. Microsoft Surface And NUI Microsoft Surface computing uses sensing and display technology to infuse everyday surfaces with digital content and is comprised of the following unique attributes: Object recognition Direct interaction Multi-touch Multi-user Object recognition Direct interaction Multi-touch Multi-user

5. How Surface Is Being Used Today • 5 Verticals • Automotive • Financial Services • Healthcare • Leisure and • Entertainment • Retail

6. announcing… Microsoft Surface Hardware And SDK availability

7. Developing ForMicrosoft Surface  Robert Levy Program Manager Surface SDK  Doug Kramer Lead Developer Surface SDK

8. demo Visualizing Surface Input

9. Congratulations! You already know the basics of building applications for Surface

10. Surface v1 Architecture Surface Applications WPF APIs Core API Shell UI & APIs Vision System Windows Integration Windows Vista SP1 Microsoft Surface Hardware & Drivers

11. Surface And WPF • Expose Surface capabilities in a way consistent with WPF • Infrastructure integration for consistent behavior with mouse • Automatic hit testing to find the control being touched • Sets attached properties on controls (IsContactOver, etc.) • Raises routed events on controls (ContactDown, etc.) • Custom versions of most built-in WPF controls • Custom controls for Surface-specific UX • Base classes for building custom Surface controls

12. Surface API Usage With WPF XAML <Window> <Canvas Mouse.MouseDown=“OnMouseDown“> <Button Click=“OnButtonClick”/> </Canvas> </Window> Code void OnMouseDown(object sender, MouseEventArgs e) { // ... } void OnButtonClick(object sender, RoutedEventArgs e) { // ... } • XAML • <s:SurfaceWindow> • <Canvas s:Contacts.ContactDown=“OnContactDown“> • <s:SurfaceButton Click=“OnButtonClick”/> • </Canvas> • </s:SurfaceWindow> • Code • void OnContactDown(object sender, ContactEventArgs e) • { // ... • } • void OnButtonClick(object sender, • RoutedEventArgs e) • { // ... • }

13. Surface Classes For WPF Input Mouse Stylus MouseEventArgs StylusEventArgs Tablet TabletDevice MouseDevice StylusDevice

14. Surface Versions Of Common Controls SurfaceWindow Full screen on the Surface Oriented towards the user SurfaceMenu SurfaceContextMenu SurfaceMenuItem Multiple menus can be used at the same time SurfaceButton SurfaceRadioButton SurfaceCheckBox SurfaceToggleButton Only “click” when all contacts are removed

15. Surface Versions Of Common Controls • SurfaceListBox • SurfaceScrollViewer • Determines scrolling vs. select • Allows flicking • Provides “elastic” effect • SurfaceSlider • SurfaceScrollBar • Allows flicking • Track input results in smooth animation and capture • Subtle UX when not in use • SurfaceTextBox • SurfacePasswordBox • Invokes virtual keyboard oriented towards user when touched

16. Neat, But Does It Justify Buying A Surface? • We’ve made it easy to port WPF and XNA apps to Surface • Great for developers • Re-use code, skills, and tools • Get started quickly • …but little value to users

17. Is this really just a big PC? I don’t like having to share and wait my turn Constantly reaching across the table is not fun Reading upside down makes my head hurt

18. Making Surface Shine • Great apps leverage all of what makes Microsoft Surface unique • Multi-touch • Multi-user • 360° UI • Gestures and manipulations • Interaction with physical objects • WPF and the Surface SDK make this easy

19. ScatterView By Example

20. Introducing The ScatterView Control • Quickly enables 360°, multi-user apps • Gives users direct, shared control of layout • Standardizes manipulations for Resize, Move, and Rotate • Same familiar programming model as ItemsControls, ListBox, TreeView, Menu, etc.

21. demo ScatterViewIn Action

22. Recognizing Tagged Objects Byte Tags Contact.Tag.Byte 256 unique values Contact.Tag.Identity >340,282,366,920,938,000,000,000,000,000,000,000,000unique values Identity Tags

23. Introducing The TagVisualizer Control Robert Levy Send Email Simplifies common scenarios involving tagged objects • Displays interactive UI under, around, or next to objects • Keeps UI in sync with movements • Handles temporary lifting TagVisualizer Control UI for each object Tag values to look for Positioning of UI relative to tag Max # of each object allowed How to handle tag removal

24. demo TagVisualizerIn Action

25. Is this really just a big PC? I don’t like having to share and wait my turn Constantly reaching across the table is not fun Reading upside down makes my head hurt

26. Controls For Making Surface Shine Robert Levy Send Email ScatterView • Multi-touch • Multi-user • 360° UI • Natural interactions with virtual objects TagVisualizer • Object recognition • Natural interactions with physical objects

27. demo Redesigning An App For Surface

28. Touch Development Roadmap Windows 7 Release NET 3.5 / Surface 1.0 Release NET 4.0 / Surface 2.0 Release WPF Application Native Win32 Application WinFormsApplication Surface Application Surface SDK 1.0 Surface SDK 2.0 Multi-Touch Controls Surface Multi-Touch Controls & API Multi-Touch API Managed Wrapper and Interop WPF 3.5 x WPF 4.0 WPF 3.5 Multi-Touch API Multi-Touch API and Controls Surface Hardware Windows Vista Surface Hardware Windows 7 Windows 7 Multi-Touch API

29. Summary • Microsoft Surface is shaping the future • Multi-touch, multi-user, 360° UI, object recognition, … • UI design requires creativity and innovation • Coding builds upon what you already know • Microsoft is creating a unified API for touch

30. Call To Action • Help us change the world • Create apps using Surface’s unique capabilities • Get Microsoft Surface developer hardware • Order forms on http://surface.com • $1,500 discount until November 15th • Swipe your badge to receive an e-mail invitation to join http://community.surface.com • Download the Microsoft Surface SDK & Simulator • Participate in online forums and training

31. Related Content • Hands on Labs • PCHOL19: Using the Microsoft Surface Simulator • PCHOL20: A Lap Around the Microsoft Surface SDK • Surface HOL: Exploring Vision Based Object Recognition with Microsoft Surface • Demo applications from partners • Scavenger Hunt game • PC03: Developing multi-touch applications with Windows 7 • Wednesday, 1:15

32. Questions?Please submit session feedback!

33. Evals & Recordings Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com

34. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. APPENDIX

37. Simulator for Off-Table Dev & Test • Replicates input • With multiple mice • With one mouse • Click to place multiple contacts • Click and drag to group them together • Move, rotate, and stretch the group • Supports all types of contacts • Finger, resizable blobs, and tags • Supports input automation for testing • Recording and playback UI • Simulator APIs 37

38. Manipulation & Inertia Processor APIs • Low level APIs for building controls with consistent behavior • Reduces the cost of building new controls • Increases consistency for end users • Interpreting Surface input as 2D transformations • Input: list of contacts to track • Output: scale, rotation, movement, velocity data (~60 FPS) • Used by: ScatterView, ScrollViewer, Slider, Concierge map • Two modes of inertia • Initial velocity + deceleration rate • Realistic • Used by ScatterView and Concierge map • Initial velocity + final position/orientation • Goal-based • Used by ScrollViewer and Slider

39. Ups & Downs Mouse.MouseDownMouse.MouseUp • Contacts.ContactDownContacts.ContactUp

40. Up & Down Event Comparison • Similarities: • Naming convention & meaning • Routing strategy • Tunneling (“preview”) events & bubbling events • Events stop once marked as handled • Hit testing & capture logic • Differences: • Need to find out which contact the event is for • New Contact object is created with each Down event

41. Movements Mouse.MouseMove … Mouse.MouseMove … Contacts.ContactChanged … Contacts.ContactChanged …

42. MouseMoved == ContactChanged? ContactChanged includes: • Position • Orientation • Recognized type • Size • Shape • Area Wait a sec… he said this was supposed to be the same as mouse. Why are these events named differently?

43. Movement Event Comparison • Similarities: • Routing strategy • Tunneling (“preview”) events & bubbling events • Events stop once marked as handled • Hit testing & capture logic • Event for changes in position • Differences: • ContactChanged is not just for movements • Change events only happen between Down & Up • Need to find out which contact the event is for

44. Position Contact.GetPosition(UIElement) • Relative to specified UI element • GetPosition(window) = {50, 100} • GetPosition(button) = {10, 10} • To track position changes across multiplecontrols, get position relative to a common parent

45. Orientation 270° Contact.GetOrientation(UIElement) • Degrees clockwise from relative east • Range and usefulness varies • Tagged objects: 360° • Generic blobs: 180° • Fingers: 360° but…unreliable when straight down 0° 180° 90°

46. Ellipse & Bounds RectContact.BoundingRect Ellipse Contact.GetEllipse(UIElement) void Contact.UpdateEllipse(Ellipse, UIElement) • Ellipse is returned with transforms that counteract transforms on the specified element • Adding the shape as a child element results in UI matching the physical object • Bounding rectangle is always screen-aligned • UpdateEllipse method can improve performance