1 / 79

High Performance Pen Interfaces

Ken Hinckley Microsoft Research October 22 nd , 2004 kenh@microsoft.com http://research.microsoft.com/~kenh. High Performance Pen Interfaces. Pen Computing Now?. Project #1: Stitching. Ken Hinckley Gonzalo Ramos Francois Guimbretiere Patrick Baudisch Marc Smith. Wireless Networks.

piper
Download Presentation

High Performance Pen Interfaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Ken Hinckley Microsoft Research October 22nd, 2004 kenh@microsoft.com http://research.microsoft.com/~kenh High Performance Pen Interfaces

  2. Pen Computing Now?

  3. Project #1: Stitching • Ken Hinckley • Gonzalo Ramos • Francois Guimbretiere • Patrick Baudisch • Marc Smith

  4. Wireless Networks • Still an immature technology – • Flaky, hard for users to configure, slow, … • Most research has focused on small-screen presentation of web pages • What are the completely new & unique affordances of wireless connectivity? • Odd phrase: not what it is, but what it lacks

  5. Wireless Network 2004 = Horseless Carriage 1904? • Are Wireless Networks really just wired networks without the wires? • Or are they something completely different?

  6. Wireless Device Soup: Which links does the user want?

  7. Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Name That Device

  8. Pen stroke that spans displays Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Stitching

  9. How is a connection established? What type of connection is required? What information is shared? What is the spatial relationship between devices? How do users share physical space? Explored in context of photo sharing app Central Design Questions

  10. AVI’04 ACM Advanced Visual Interfaces 2004 VIDEO

  11. Recognizes timing & dynamics of pen trajectory There is nothing special about the pen! Wireless signal strength determines nearby devices Tablet 2 Tablet 1 Δt 1. Establishing a Connection

  12. User selects a multi-device command e.g., Pie menu at end of stitching gesture 2. Type of Connection

  13. Many commands may need operands Lasso to select, continue to other display to stitch. 3. What Information to Share?

  14. Fits a line to absolute coords of user’s pen stroke 4. What is Spatial Relationship: Automatic Screen Calibration

  15. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 5. How Do Users Share Physical Space?

  16. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 6. How Do Users Share Physical Space? • Homework assignment: • Sit right next to someone at airport (when it is not necessary to do so) • Time how long it is before the other person leaves

  17. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 5. How Do Users Share Physical Space?

  18. Don’t require contact : touching is taboo “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching

  19. Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Gives users flexibility to be involved, or not: mood, task, … Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Proxemic Lessons for Stitching

  20. Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Gives users flexibility to be involved, or not Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Support varying relative body orientation Proxemic Lessons for Stitching

  21. 6 conversations 3 1 0 Proxemics: Orientation of Spaces • Communication patterns change wrt orientation • Stitching supports: • Shoulder-to-shoulder • Face-to-face • At 90 angles Relative frequency:

  22. Multiple Devices: Cooperative Stitching User 1 User 2 User 4 User 3 Ongoing work

  23. Multiple Devices: Cooperative Stitching User 1 User 2 User 4 User 3 Ongoing work

  24. Project #2: Scriboli • Ken Hinckley • Patrick Baudisch • Gonzalo Ramos • Francois Guimbretiere

  25. Round trip problem on Tablet PC • 4 trips (+ some xtra) for copy/paste

  26. Round trip problem • 4 trips (+ some xtra) for copy/paste

  27. Round trip problem • 4 trips (+ some xtra) for copy/paste

  28. Round trip problem • 4 trips (+ some xtra) for copy/paste

  29. Round trip problem • 4 trips (+ some xtra) for copy/paste

  30. Round trip problem • 4 trips (+ some xtra) for copy/paste

  31. Round trip problem • 4 trips (+ some xtra) for copy/paste

  32. Round trip problem • 4 trips (+ some xtra) for copy/paste

  33. Round trip problem • No right click, kbd shortcuts to avoid this!

  34. GEdit – Kurtenbach & Buxton Delete one obj. Copy – add final “C” to move Delete group – end stroke inside lasso • One of the first works considering pen phrases • Every stroke is gesture • Only 3 cmds • Group vs. single obj. inconsistent Move – end stroke outside. Objects move on PenUp.

  35. SKETCH – Brown University • Adding 1 more gesture breaks everything • Specific to 3D, difficult to learn how to use

  36. Marking Menus • Prompted BlindScale independence • Self-revealing: prompts w/ available cmds • Rapid expert use: blind marking • Menu appears only after 1/3 second delay • Size of mark does not matter

  37. FlowMenu – Francois Guimbretiere • Integrates command selection with continuous parameter manipulation • But no “blind marking,” size of gesture matters Crossing inout chooses Item… submenu Crossing outin chooses Move command Dragging continues

  38. Tivoli • Tivoli project @ Xerox PARC • We wanted quick, informal feel of scribbling

  39. Learning Cursive • building blocks of writing as basis for pen UI? • not natural handwriting recognition

  40. The Deep Structure of Handwriting • The “scribble” of Scriboli • Phasic muscle contractions: • fast, brief, usually repetitive • Pre-programmed, internally driven • “The conversion of those […] O’s to real words and sentences happens when small variations in the amount of verticality or laterality […] are introduced into the movement. The extreme orderliness and predictability of individual handwriting is due to this mode of generating and controlling the movement.”F. R. Wilson, The Hand, p. 163

  41. Design Philosophy of Scriboli • RAPID • Fast motions, repeatable, no waiting / dwelling, minimal demands on visual attention • UNAMBIGUOUS • No guessing games • No reco unless I tell the system otherwise • (There is almost no reco in Scriboli) • EXPRESSIVE • Supports many commands, easy to grow design • Mechanisms for a wide span of pen apps, not restricted to specific niche application

  42. Punct. space capitals period comma object verb indirect obj. Written text separates words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase Grammar for pen input • Scriboli proposes fundamental building blocks • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases

  43. Ink vs. Gesture • Who does the recognition? • Ink: strokes intended primarily for human • Gesture: strokes intended for computer • Virtually all pen systems make this distinction at some level • Very seductive to attempt to do automatically • Many systems spend most or all effort just on this • Any pen gesture might occur in freeform input • Scriboli uses explicit ink/gesture mode switch

  44. Ink/Gesture study w/ U. Washington • 5 Techniques: • Barrel Button • Tap+hold • NPH button • Pen Pressure • Pen Eraser

  45. Punct. space capitals period comma object verb indirect obj. Written text separates words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase Grammar for pen input • Scriboli proposes fundamental building blocks • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases

  46. Delimiters: Single-stroke lasso selection + marking • Initial part of stroke is the scope • Pigtail is the delimiter • Tail interpreted as the mark

  47. What is a Delimiter? • A delimiter is “something different” about the pen input stream • Determines lexical structure of pen stroke • Pigtail only intended as delimiter during Gesture mode • Not an automatically inferred mode switch

  48. Punct. space capitals period comma object verb indirect obj. Written text separates words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope(lasso,line, …) Marking menu Crossing manip. phase Grammar for pen input • Scriboli proposes fundamental building blocks • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases

More Related