Gesture Toolkits

1. Talk v1.0 Gopi Flaherty April 15, 2003 Gesture Toolkits

2. Graffiti on Palm Extensions for X11 Windows for Pen Computing Boring Gesture Systems

3. Keyboard-like behavior Simple input model with streams of characters Most applications behave as if they have a keyboard Some Palm devices have keyboards instead of graffiti - one even comes in a graffiti or keyboard version - and apps don�t notice Graffiti

4. Graffiti 2.0: different alphabet, more two stroke characters When the user enters a double stroke gesture, the app receives a letter, a backspace, and the next letter API documentation suggests hardcoding the list of these gestures, and implementing a delay if this behavior is a problem Graffiti

5. Simple extension to X11 Has a configuration file with a 3x3 grid. Gestures are defined based on the order you hit squares in the grid Behaves similarly to Graffiti Toolkit? xstroke

6. Gestures for simple commands Some simple gesture systems - possibly graffiti - let you have gestures for commands Generally very limited, also requires that gesture system knows about commands, and most simple gesture systems don�t get very involved Slightly more advanced

7. Existing apps - console UNIX apps via graffiti or xstroke is better than trying to make input fields Retrofitting an app such that half of it is gesture-based frequently has problems Good pen/gesture input requires fundamentally different ways of interacting and programming Are these approaches ever useful?

8. Systems that provide some gesture resources but only at a more basic level Useful widgets, but with only limited configurability Usually a fixed set of gestures that are recognizable Middle of the road

9. Text entry widget is very configurable Phone, Date, Time, Address and Name are field types Newton Widgets

10. Text entry widgets have many callback methods that can be utilized for more configurability viewWordScript is called when a word is recognized viewChangedScript is called to indicate that the text in the entry has changed viewGestureScript indicates that a gesture has been entered - this would be a raw, unrecognized gesture

11. Gesture widgets are primarily forms-based Internal implementation is opaque Application isn�t involved in recognition normally Becoming slightly more involved in gesture processing appears to be possible through some reverse engineering

12. Why do we need toolkit support? There are hundreds of useful apps for the Newton Most apps seem quite content to use the standard widgets Newton widgets are clearly optimized for pen and gesture input, despite not being very involved in the gesture parsing The Interesting Stuff

13. Fundamentally, the widgets are still very much forms-based Intelligent dynamic forms can be extremely interactive, but they are still forms and still limited Hacking widgets to do things they weren�t intended to do can get tedious and slow

14. What is the cursor? Write gestures centered on cell Edit line on the top Early newton app: viewGestureScript -> viewWordScript? Common apps: spreadsheet

15. Standard text entry fields format text themselves, with limited flexibility Even basic word processors have quite complex custom formatting needs - justified text, tab editing, headers and footers It can be hard to combine custom rendering code with standard gesture widgets Common apps:Word Processor

16. If a toolkit is to provide easy support for complex apps, it needs to be extremely transparent Applications must be able to override as much or as little as is necessary for them to achieve their UI goals Results:

17. Hybrid system, designed to allow handwriting in apps without modification Many basic recognition features can be used in apps that work with or without the pen Provides more advanced features and tighter pen integration that is harder to integrate with non-pen interaction Microsoft TabletPC

18. PenInputPanel

19. PenInputPanel Object that can be attached to any input field in your app Allows app to override default behaviors

20. Superclass of RichEdit or RichTexBox Provides far more control Applications can register controls to receive various predefined gestures InkEdit

21. Pre-Defined Gestures

22. Future Gestures 1

23. Future Gestures 2

24. Custom Recognizers Custom Recognizer objects can be registered to Ink objects An InkCollector object collects gestures, which can be passed to a custom recognizer, or to the original Microsoft one, or both in parallel

25. For each recognizer:

26. Egg Freckles? One of the largest challenges with computer recognition of any type Dealing with recognition ambiguity can be the difference between a pen system working or not working Recognition Ambiguity

27. Sidesteps the issue completely You write the stroke the way you�re supposed to If it mis-recognizes, the assumption is that it was your fault Blaming the user is argued by some to be the cause of its success Graffiti:What ambiguity?

28. Basic input: What letter or word did the user type? One of the easier types of ambiguity to deal with The recognizer provides an n-best list, select the top, and you provide a UI to making alternate selections from the list Different kinds of ambiguity

29. This can be somewhat more challenging with multiple recognizers: how do you intermingle their respective n-best lists? If gestures can perform actions, the UI can be difficult: how do you present a choice of �cut�, �copy� or the word �hello� to the user? Newton had interesting choice in n-best: do letter by letter, ignoring dictionary weighting. Produced a new n-best list Simple Ambiguity

30. If gesture recognition is being used in a time-constrained environment, it brings up other interesting challenges Resolving ambiguities is distracting and slow, but retaining n-best information indefinitely can increase storage space immensely Lazy ambiguity resolution can�t be used for action gestures without extreme penalties Ambiguity and time

31. Later versions of the Newton OS let you do lazy recognition When text is written, it�s retained as the original glyphs, segmented into words and reflowed with word wrap as normal text At any stage in the future, the user can request that text be recognized Retain as glyphs is also an option in the n-best correction widget The lazy approach

32. Very lazy recognition Somewhat similar to the Newton�s �keep it as glyphs�, but significantly more structured than simple text editing One of the design goals is to deal with the problem of recognition systems demanding immediate ambiguity resolution - very distracting DENIM and SILK

33. What about segmentation? Most shipping recognition systems do segmentation of strokes into individual collections, and do not allow any future re-analysis of the decision Allowing segmentation ambiguity is difficult from a UI perspective - double tapping a word is easy, but what about trying to resolve a pair of words Meta-ambiguity

34. Another aspect - who gets the gesture? On a screen with forms, how do you resolve which text entry field gets the text? Approaches I have seen are the simple ones - there�s a bounding box, strokes in the box are for that widget This reduces the density of widgets and requires careful design Segmentation

35. A common problem in poorly designed Newton apps: text entry box right next to control widgets Write a stroke - start too close to the window drag area and the window moves instead Nearly impossible to resolve, since control widgets demand immediate feedback, but a serious problem on small screens Was it a gesture?

36. How much support is needed in the app for dealing with ambiguity? Cut and paste on the Newton retains n-best objects In theory, nearly infinite knowledge possibilities Which text field wants the input? Application Support

37. Imagine two text fields: One is a city, the other is a county It�s generally impractical to have a complete list of all of these, but a possible list can exist If there is ambiguity about which field an entry is for, the application�s knowledge about likely values can add increased weighting knowledge Application Intelligence

38. Dean Rubine Specifying Gestures By Example

39. Provide a series of samples for each gesture Works exclusively with single strokes Eager: recognizes during the stroke to increase responsiveness Drawing ink on screen is easy - so process during the stroke Battery systems slow processor during strokes Basic outline:

40. Jennifer Mankoff, Scott E. Hudson, Gregory D. Abowd Providing Integrated Toolkit Level Support for Ambiguity in Recognition Based Interfaces

41. Basic ambiguity

42. Segmentation Ambiguity

43. Shortcomings in other systems Nobody seems to deal with target ambiguity Segmentation ambiguity is also rarely tackled Application-specific knowledge is rarely used

44. Toolkit identifies ambiguous events and sends them to the mediation subsystem Each mediator receives the event in turn Mediators can ignore, partially resolve, or defer their decision to the future Some mediators have a UI, some do not Also allows much lazier mediation Mediation Toolkit