Scriboli: High Performance Pen Interfaces

1. Scriboli: High Performance Pen Interfaces Ken Hinckley Patrick Baudisch Gonzalo Ramos Francois Guimbretiere Microsoft Research

2. Scriboli Design Goals • Speed without keyboard hotkeys • Cognitive footprint diminishes with use • Recall / declarative  procedural skill • Minimal demands on visual attention • Repeatable motions for the user • Predictable & dependable system behavior • Expressiveness • Support many command structures & apps • Economy of design • only a few things to learn

3. SKETCH – Brown University • Adding 1 more gesture breaks everything • hard to learn, gestures specific to 3D editing

4. Recognize Pen Gestures? • Classic problem: “ink” or “gesture” ? • Hard problem: any ‘gesture’ could be ink • Inferred decision comes too late • real-time interactive feedback difficult • Make it a physical skill • All we need to solve this is ONE BIT of info • physical button on bezel • habit-forming, deterministic, low attentional demand • nonpreferred hand, no trip to “lasso mode”

5. Phrase Structure for Scriboli Scope Command Parameters Separator (start) Separator Scope/Command Separator Command/Parameters Separator (Done)

6. Phrase Structure for Scriboli Scope Command Parameters Separator (start) Press “Gesture” Separator Scope/Command Separator Command/Parameters Separator (Done) Fast and reliable [Sellen 90, 92] Can be chunked by users

7. Phrase Structure for Scriboli Scope Command Parameters Separator (start) Press “Gesture” Separator Scope/Command Pigtail Separator Command/Parameters Separator (Done) Fast, based on muscle “memory” Reliable and easily chunked by users

8. Phrase Structure for Scriboli Scope Command Parameters Separator (start) Press “Gesture” Separator Scope/Command Pigtail Separator Command/Parameters Crossing boundary Separator (Done) • Merge command selection and direct manipulation • FlowMenu [Guimbretiere 00], Control menu [Pook 00]

9. Phrase Structure for Scriboli Scope Command Parameters Separator (start) Press “Gesture” Separator Scope/Command Pigtail Separator Command/Parameters Crossing boundary Separator (Done) One stroke for scope, command, and parameters Non-modal system (quasi-modal)

10. Delimiters for scope selection + marking • A delimiter is “something different” • lexical structure of stroke • e.g. self-crossing gesture stroke • DEMO / VIDEO :delimiter techniques

11. Results – Completion Time • Left bar – main block; right bar – repeated block • Dotted bars are Pigtail2 design iteration Button Handle Pigtail Pigtail2 Timeout

12. Results – Learning Effects • For the 24 trials of Repeated Invocation Block

13. Results – Error Rate • Error rate of selecting wrong marking direction Button Handle Pigtail Pigtail2 Timeout

14. Punct. space capitals period comma object verb indirect obj. Written text separate words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase Summary: Grammar for pen input • Scriboli proposes fundamental building blocks • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases

15. Stitching: Pen Gestures that Span Multiple Displays Microsoft Research Ken Hinckley Gonzalo Ramos Francois Guimbretiere Patrick Baudisch Marc Smith

16. Wireless Network 2004 = Horseless Carriage 1904? • Are Wireless Networks really just wired networks without the wires? • Or are they something completely different?

17. Wireless Device Soup: Which links does the user want?

18. Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Name That Device

19. Pen stroke that spans displays Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Stitching

20. AVI’04 ACM Advanced Visual Interfaces 2004 VIDEO

21. Recognizes timing & dynamics of pen trajectory There is nothing special about the pen! Wireless signal strength determines nearby devices Tablet 2 Tablet 1 Δt Establishing a Connection

22. Fits a line to absolute coords of user’s pen stroke Automatic Screen Calibration

23. SociologyProxemics: How people use space Invisible bubble that surrounds an individual How Do Users Share Physical Space?

24. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 6. How Do Users Share Physical Space? • Homework assignment: • Sit right next to someone at airport (when it is not necessary to do so) • Time how long it is before the other person leaves

25. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 5. How Do Users Share Physical Space?

28. Don’t require contact : touching is taboo “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching

29. Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Gives users flexibility to be involved, or not: mood, task, … Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Proxemic Lessons for Stitching

30. Multiple Devices: Cooperative Stitching User 1 User 2 User 4 User 3 Ongoing work

31. Multiple Devices: Cooperative Stitching User 1 User 2 User 4 User 3 Ongoing work

32. Questions?

33. Handle vs. Pigtail • Handle can get in the way • Adds some visual clutter • Must check to be sure landed on handle • Pigtail handles more than one scope elegantly • Self-referential gesture