Natural User Interfaces & Microsoft Surface Development

1. Natural User Interfaces & Microsoft Surface Development

2. Why are we here? • Obligatory corporate plug • Obligatory personal plug • The real content: • What is a Natural User Interface (NUI)? • What is the Microsoft Surface? • MS Surface development environment • Demos and code: • Hello, Surface! • Raw Images in WPF • Contact Data Visualization • Surface Controls • Tag Visualizations • The Future: Windows Touch and Surface 2.0

3. Based on our philosophy of The Intelligent Application of TechnologyTM, we design, develop, and implement technologysolutions that help our customers run their businesses better. Our Company, Culture and Success

4. T4G at a glance 1996 Year T4G was established 3 • # of consecutive years on the Great Place to Work® list 260+ 80 # of employees in 7 locations across North America % of customers in 2010 who invited us back for multiple projects

5. OUR VALUES Integrity Passion Communication Teamwork Adaptive Innovation Entrepreneurialism Value Conscious Social Responsibility

6. Delivering leadership and expertise Our Vertical Expertise • Our Supporting Services • Business Intelligence • Infrastructure & Security • Software Development & Integration • Content Delivery • Mobile Solutions • Social Media Strategy & Measurement • Strategic Planning • Hosting & Application Management • Portals & Collaboration • User Experience & Design Travel and Hospitality Retail Energy Financial Services Professional Services Healthcare Government Communications

7. Who am I? • Principal Consultant at T4G • Focused on Portal Technologies (SharePoint) • Also work on NUIs and Multi-touch technologies, among other things • 25+ years on software and various technology things

8. What is a Natural User Interface? A natural user interface is a user interface designed to use natural human behaviours for interacting directly with content.

9. A New Way of Interacting Speech Social Interaction Eye Tracking Augmented Reality NUIis much more than Touch Gestures Natural Learning User eXperience Object Recognition

10. A New Way of Interacting Social Immersive “The content is the application”

11. Implications PervasiveUIs Don’t replace human interaction, enhance it Extend the real world, blur the lines between UI and real life

12. What is the Surface? “Microsoft Surface is a revolutionary multi-touch computer that responds to natural hand gestures and real-world objects, helping people interact with digital content in a simple and intuitive way.”

13. Surface is not just multi-touch • Windows 7 & WPF 4 are a great baseline for touch • Surface creates unique opportunities with specialized HW • Robust top • Horizontal form factor • Infrared camera array

14. Surface: Beyond Multi-Touch • Massive multi-touch • Multi-user experiences • Object recognition • Optical engineering

15. Surface: Multi-User • Trend: Software continues to reduce the need for face-to-face human interactions • Problem: Happiness and creativity depend on social interactions • Surface brings people together • Industrial design encourages users to gather • Input capabilities allow simultaneous use • Apps facilitate collaboration / competition

16. Surface: Massive Multi-Touch • Surface can efficiently track 50+ simultaneous fingers and objects

17. Surface: Objects • Uniquely identify & react to billions of tagged objects Byte Tags 256 unique values Identity Tags >340,282,366,920,938,000,000,000,000,000,000,000,000unique values • Tags include orientation indictors dots

18. Surface: Optics • Determine the size and shape of every input • See into and project onto physical objects

19. What makes a GOOD Surface Application? • The app must be right for the Surface paradigm • It should be social • Should not involve large amounts of “traditional” content • Needs to be implemented thinking about touch

20. Who is using it? • Retailers (Microsoft, Wind Mobile, etc.) • Hospitality (Hard Rock Café, Rio iBar, Sheraton, etc.) • Museums (Smithsonian) • Media (MSNBC) • Healthcare, Education, Others

21. Development Environment Architecture of the Surface Development Platform

22. Development Environment All stuff you know… • Windows Vista 32-bit • .NET 3.5 • WPF or XNA

23. Development Environment Get your own Surface… On a unit, you can use the Microsoft Surface SDK to develop and test Microsoft Surface applications directly. You can run the unit in two modes. In administrator mode, you can access all of the Windows Vista functionality. User mode demonstrates how the unit appears in a commercial venue. That is, the unit suppresses the Windows Vista user interface, automatically starts required applications, and makes sure that those applications are restarted if they need to. Typically, you develop applications in administrator mode.

24. Development Environment If you do not have a Surface or you share one)… On a separate workstation, you can install the Microsoft Surface SDK and use the Surface Simulator tool to simulate a Microsoft Surface test environment. • Runs the same Surface Shell as the Surface does • Use 1 or more mice to simulate multi-touch • Tools for simulating Tags or Blobs • Record and Replay interactions to support testing and debugging Limitations: • No cameras (obviously) so if you use raw image data, cannot test it on simulator • Should have a pretty good sized screen • The SDK is only supported on Windows Vista Business, 32-bit! But, there are ways around that! (http://www.brianpeek.com/blog/archive/2009/03/10/install-the-surface-sdk-on-windows-7-and-or-x64.aspx)

25. Surface SDK Features for Multi-Touch • Common controls optimized for Multi-Touch • Controls design primarily for Multi-Touch • Essential Multi-Touch UX Functionality

26. Lessons Learned • There is a lot to learn about NUIs • There is just as much to unlearn • Forget most of what you know about what makes a “good” UI • Visuals/graphics/interaction design more important than code • Less is more • Development is easier on a workstation

27. Hello, World! Plain Old WPF XAML <Window x:Class="WpfApplication1.MainWindow“ xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentation xmlns:x=http://schemas.microsoft.com/winfx/2006/xaml Title="MainWindow" Height="350" Width="525"> <Grid> </Grid> </Window>

28. Hello, World! Surface XAML <s:SurfaceWindow x:Class="HelloSurface.SurfaceWindow1“ xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentation xmlns:x=http://schemas.microsoft.com/winfx/2006/xaml xmlns:s=http://schemas.microsoft.com/surface/2008 Title="Hello Surface"> <Grid> </Grid> </s:SurfaceWindow>

29. Code Sample 1 Hello, Surface! • Basic Surface Application • Responding to Touch Events • Responding to multiple Touch Events • Handling orientation information

30. Code Sample 2 Contact Data Visualization • Types of Contact events Surface sees (fingers, tags, and blobs) • Information about contacts

31. Code Sample 3 Surface Controls • Common controls optimized for Multi-Touch • Controls design primarily for Multi-Touch

32. Code Sample 3 Surface Controls • Common controls optimized for Multi-Touch • Controls design primarily for Multi-Touch

33. Code Sample 3 Surface Controls • ScatterView Control • Adding ScatterViewItems in XAML • Adding ScatterViewItems in code (in response to Contact Events) • ScatterViewItems through Binding (with simple item template) • Handling contact events when ScatterViewItem content is “active”

34. Surface: Objects • Uniquely identify & react to billions of tagged objects Byte Tags 256 unique values Identity Tags >340,282,366,920,938,000,000,000,000,000,000,000,000unique values • Tags include orientation indictors dots

35. Code Sample 4 Tag Visualizations • Adding a Tag Visualizer to recognize a tag • Extracting the information in the tag • Displaying a Tag Visualization

36. Touch Development Roadmap Windows 7 (2009) Native Win32 Application WinFormsApplication WPF Application Surface Application Surface SDK 1.0 Multi-Touch Controls Multi-Touch Controls & API WPF 3.5 SP1 Managed Wrapper and Interop Surface Hardware Windows Vista Windows 7 Multi-Touch API

37. Touch Development Roadmap .NET 4.0 & Surface Toolkit (Q1 2010) Native Win32 Application WinFormsApplication WPF Application Surface Application Surface SDK 1.0 Surface Multi-Touch Controls &API Multi-Touch Controls & API Multi-Touch Controls Surface Toolkit for Windows Touch Managed Wrapper and Interop WPF 3.5 SP1 WPF 4.0 w/ Multi Touch APIs Surface Hardware Windows Vista Windows 7 Multi-Touch API

38. Touch Development Roadmap Surface 2.0 Native Win32 Application WinFormsApplication WPF Application Surface Application Surface SDK vNext Surface Multi-Touch Controls &API Multi-Touch Controls Surface-specific Controls & API Surface Toolkit for Windows Touch Managed Wrapper and Interop WPF 4.0 w/ Multi Touch APIs Surface Hardware Windows 7 Windows 7 Multi-Touch API

39. Surface 2.0 Integration & Extension of WPF 4 • All input is routed through the WPF InputManager • Surface input provider can send to multiple windows • Surface extension methods provide access to additional input data (tag values, orientation, shape, etc) Windows Touch input provider Manipulation & Touch Events WPF InputManager Apps & Controls Surface input provider Surface extension methods

40. Surface 2.0 • Announced at CES 2011 • Available later this year • SDK Available April 2011 (?) • Embedded Windows 7 Professional 64-bit • .NET Framework 4.0 • Windows Presentation Foundation (WPF) 4.0 • Microsoft XNA® Framework 4.0 • Windows PowerShell and DMTF DASH support, and enhanced administrator tools

41. Resources On the web: Surface Web Site (http://www.surface.com) Deconstructing the NUI (Joshua Blake, Surface MVP) My Blog (http://fyeomans.com) On Twitter: @Surface – the official MS Surface twitter identity #Surface @fyeomans

42. Questions?