Interaction: gestural input, popup widgets, interaction techniques, and hardware input devices

1. Interaction: gestural input, popup widgets, interaction techniques, and hardware input devices Prof. Michael McGuffin

2. Gestural Input • Gestures can be executed rapidly, and can be used as symbols to activate commands or select objects • Sketches • can be used to quickly enter text or diagrams, without requiring the user to switch from the pointing device to something else • Have an informal, loose, implicit structure • Can carry much more (implicit) information than typed text • Gestures as characters to recognize • Graffiti, Unistroke

3. Gestural Input Kurtenbach and Buxton (video) • Example gestures: circle objects, drag to move, or drag + draw a "C"  to copy • Such a gesture selects the noun (objects) and verb (operation) in a single action, which may be more natural, fluid, or faster than doing selection + Ctrl-C/Ctrl-X + Ctrl-V

4. Gestural Input "Teddy" (Igarashi et al. 1999). See Java applet at Takeo Igarashi’s website.

5. Phun (≈2008) • A free game/physical simulator that uses gestural input • http://www.acc.umu.se/~emilk/

6. Phun

7. Alvarado and Davis 2001 • http://www.youtube.com/watch?v=NZNTgglPbUA

8. LaViola 2007 http://doi.acm.org/10.1145/1281500.1281558

9. Observation • Most user interfaces that use gestural input require the user to first learn the gestures before they can be used. This can be a significant obstacle for novice users. • In the following slides, we will see a technique called Marking Menus that eliminates this problem.

10. Modes, and some examples of interaction techniques using gestural input or other input

11. Interaction Techniques • “are ways to use input devices to enter information” • "are made up of single input-device actions.” (Foley, et al. “Computer Graphics: Principles and Practice”)

12. Verbs(actions,commands,tools,operations) Nouns(objects,locations)

13. Verbs in a pulldown menu Nouns(objects,locations)

14. Verbs ina popup menu Nouns(objects,locations)

15. Summary • Modes can lead to mode errors, where the user thinks they are in one mode but are really in another • Visual feedback (example: changing cursor shape) to indicate the current mode is good, but often not enough to avoid mode errors • Popup menus help… • Avoid mode errors(via transient modes, and kinaesthetic feedback where the user feels pressure through their finger from holding down a button) • Increase screen space devoted to content, because the menu is only displayed when desired • Decrease the distance travelled by the cursor, because there is no back-and-forth motion between the workspace and peripheral interface elements • Create a tighter integration, or fusion, of the selection of noun and verb, leading to better mental chunking (Buxton 1986)

16. Given that popup menus have all these advantages, can we design other widgets that are even better?

17. Radial Menu (or “Pie Menu”)

18. Demo: Radial Menu in SimplePaint

19. An interface that uses, effectively, radial menus Yatani et al., CHI 2008

20. Radial menusversus linear menus

21. Radial menusversus linear menus

22. Radial menusversus linear menus

23. Radial menusversus linear menus

24. Radial menusversus linear menus

25. Radial menusversus linear menus Directions are easier to rememberand reproduce than distances

26. Hierarchical Radial Menu

27. "Mouse Gestures" for Firefox

28. Marking Menu • “Scale invariant recognition”: The recognition of the gestures (marks) does not depend on the length of the segments; only the angles matter. Hence, the marks can be drawn small and quickly, or ballistically. • A user who knows what mark to draw doesn’t even need to see the menu popup.

29. The set of gestures is discoverable and “self-revealing”, contrary to other gestural interfaces

30. Improved graphical presentation

31. Marking Menus • Video/Demonstration

32. Marking Menus Video/Demonstration

33. Marking Menus Video/Demonstration

34. Transition from novice to expert Marking Menus:Transition is gradual and natural ! Traditional menus:Pointing versus shortcuts

35. Summary Marking Menus • Can be operated faster than linear menus(ballistic, directional, scale-invariant marks) • Have a self-revealing gesture set • Enable a smooth transitionfrom novice use to expert use • Can be used to select both a noun and verb • Are limited to 8 items per submenu,and limited to a depth of about 3 levels

36. Marking Menus • Allow for faster selection than with linear menus(directional, ballistic movements) • Can be used without looking at the screen(“eyes-free operation”) • Have a discoverable set of gestures • Allow for a gradual and natural transition from novice to expert • Can be used to select a noun and verb • Are limited to approximately 8 items per submenu, and a depth of about 3 levels

37. Summary • Temporary modes, held by holding down a button or key with kinesthetic feedback … • Help avoid mode errors • Example: a key that must be held down to maintain a temporary mode change, such as the Shift key on keyboards • Example: popup widgets • Help combine noun+verb selection • Only require screen space when in use • One popup widget with many advantages: Marking Menus • Allow for ballistic gestures • Allows users to gradually transition from novice to expert

38. Other examplesof modes, tools, etc. • video: "Selection and Positioning tasks", Bill Buxton 1983

39. Some observations • Techniques shown in the video: • 1. Drag-and-drop • 2. Modal toolbar • 3. « Moving menu » (a kind of popup menu) • 4. « Moving menu » with memory • Questions: of the 4 techniques, which ones … • Are not modal (or only have a temporary mode with kinesthetic feedback), and so help to avoid mode errors ? (Answer: 1 and 3) • Avoid back-and-forth movements between the workspace and the toolbar, and do not have screenspace consumed by a toolbar ? (Answer: 3 and 4) • Have visible affordances (toolbar), therefore showing the possible operations to a novice user ? (Answer: 1 and 2) • Allow the same shape to be created many times repeatedly, very quickly ?(Answer: 2 and 4) • Ensure that each shape is always selected with the same dragging gesture, allowing the user to learn the gestures by heart and then execute them rapidly ? (Answer: 1 and especially 3.)

40. Can we extend popup menus/widgetsfor other uses?

41. Other popup menus and widgets • Hotbox (Kurtenbach et al., 1999),Control Menus (Pook et al., 2000),Flow Menus (Guimbretière et Winograd, 2000),FaST Sliders (McGuffin et al., 2002),Tracking Menus (Fitzmaurice et al., 2003),Trailing Widget (Forlines et al., 2006)Hover Widgets (Grossman et al., 2006),PieCursor (Fitzmaurice et al., 2008) • These widgets and interaction techniques are designed for: • A large number of commands • Controlling continuous variables • Input of text and numbers with gestures • Use with a stylus (pen), for example, on a “tablet PC”

42. FlowMenus (Guimbretière et al., 2000)

43. 2D manipulation with FlowMenus(Guimbretière et al., 2000)

44. The Hotbox in Maya: a 2D menu [Kurtenbach et al., 1999]

45. The Hotbox in Maya: a 2D menu [Kurtenbach et al., 1999]

46. The Hotbox in Maya: a 2D menu [Kurtenbach et al., 1999]

47. Demo of a more recent kind of Hotbox (McGuffin and Jurisica 2009) used for manipulating a network visualization