People, Pens, and Computers

1. People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com

2. New devices, old tasks Middle picture from Sellen et al.

3. People, Pens, and Tablet PC The New Yorker Illustration from Ken Hinckley presentation at Stanford

4. Typical setting for today’s interface • Fixed stable environment, with a keyboard, • Indirect interaction, • High precision pointing system

5. Typical Tablet PC use • Portable, unstable environment, without a keyboard • Direct interaction, • Low precision aiming

6. Document area How to create a more fluid interface? Interface framework How to make it more pen-friendly? Pen based interface for Tablet PC

7. Interface framework How to make it more pen-friendly? Pen based interface for Tablet PC

8. CrossY: crossing-based GUI[Apitz & Guimbretière 04] • Empirical foundations • Use of strokes to cross target is more pen friendly • Crossing is as efficient as point-and-click [Accot & Zhai, 2002] • The basic interactor • How expressive is it?

9. CrossY video

10. Previous Work • Theoretical basis • Steering Law, Trajectory-Based Tasks [Accot & Zhai 97-02] • Limited scope examples • Toggle Map [Baudish 98] • Lotus Notes: multiple e-mail selection • Conceptual design • Visual Instruments: [Winograd & Guimbretière 98] • Overloading • Gedrics: [Geißler 95]

11. Command composition • From stroke-by-stroke interaction • Borders are used to validate/cancel

12. Command composition • From stroke-by-stroke interaction • Borders are used to validate/cancel • To multi-command stroke

13. Scrolling Line by line area Page by page area Absolute area

14. CrossY scrollbar • Overloading simplify interactions • Shorter distances to issue commands • Not as much precision necessary

15. CrossY scrollbar • Overloading simplify interactions • Shorter distances to issue commands • Not as much precision necessary • Extending stroke for repeat • No need to wait for a timeout

16. Cursor control • Cross to jump to an absolute position

17. Cursor control • Cross to jump to an absolute position

18. Cursor control • Cross to jump to an absolute position • Near drag for coarse adjustment

19. Cursor control • Cross to jump to an absolute position • Near drag for coarse adjustment

20. Cursor control • Cross to jump to an absolute position • Near drag for coarse adjustment • Far drag for fine adjustment • Similar to FineSlider [Masui 95] • But one single stroke

21. Use of directionality • Continuous find and replace

22. Use of directionality • Continuous find and replace • Reverse direction for undo

23. What we learned • Very well received by users • HCIL open house • UIST • Space requirements • Similar to point-and-click • Trade-off with command combination due to sloppiness • Overloading vs. easy discovery • Consistency helps with adoption • Known in Windowing systems

24. Document area How to create a more fluid interface? Pen based interface for Tablet PC

25. Scriboli and Stitching Ken’s presentation

26. My desk

27. Affordances of paper documents[Sellen 01] • Easy to navigate • Two-handed interactions and tactile feedback • Reading across more than one document at once • Easy to annotate • Directly on the document or on a nearby pad • Well accepted during meetings • Socially accepted conventions • Very difficult to modify • Printed documents are created and edited as digital documents • Expensive to distribute and archive

28. Bridging the gap: previous work • Digital emulation • FreeStyle system [Wang 89] • MATE [Hardock 93] • XLibris [Schilit 98], [Golovchinsky 02] • Tight coupling • DigitalDesk [Wellner 93], Ariel [Mackay 95] • A-Book [Mackay 02] • PaperLink [Arai 97] • Intelligent Paper [Dymetman 98] • Paper as input device • Xax [Johnson 93] • Anoto • Paper PDA [Heiner 99], [Avrahami 01]

29. Cohabitation[Guimbretière 03]

30. Stroke capture • Requirement • Stroke coordinates on the page • Page ID • Large address space • Possible technologies • Anoto • DataGlyphs [Hecht 94] • MEMO pen [Nabeshima 95] From Anoto documentation

31. System architecture

32. My Desk with PADD

33. PapierCraft: command system for PADD[Liao, Guimbretiere and Hinckley 05] PADD notepad PADD document Anoto pen

34. PapierCraft: using PADD as proxies Copy Paste Synchronization On your Tablet PC

35. PapierCraft • Gesture/Ink • Use the command button • Scopes • Command selection • Marking menu • Writing down an unambiguous prefix of a command • Batch mode processing

36. Early feedback and future work • Small scale evaluation • Commands with written words were very popular • Higher recognition rate needed for strokes only feedback • Contextual information might provide enough feedback during pastes • Future work • More reliable recognition engine • Pen-based feedback • Haptic feedback, LED • Real time processing : streaming stroke

37. Conclusions • Four pen-based interfaces • CrossY explores crossing-based interfaces • Scriboli explores high performance direct manipulation interfaces • Stitching explores multi-devices interactions • PapierCraft uses PADD as document proxies • One goal • Bring the ease of use of pen and paper to digital interfaces • Future • Seamless integration of both media

38. Acknowledgments • Collaborators • Jim Hollan • Students • Georg Apitz, Nicholas Chen (CrossY/ScribolY) • Kevin Conroy, Dave Levin, Chunyuan Liao (PADD/ProofRite) • Sponsors • Microsoft • HP • Anoto, Logitech, Maxell • Colleagues and friends • Corinna Löckenhoff • Samrat (Bobby) Bhattacharjee • Ben Bederson