The Challenges and Potential of End-User Gesture Customization

1. The Challenges and Potential of End-User Gesture Customization UranOh1 and Leah Findlater2 1 Department of Computer Science 2 College of Information Studies University of Maryland, College Park uranoh@cs.umd.edu | leahkf@umd.edu

2. Touchscreen gestures are widely used… Who designs these gestures? Design experts. Apple’s touchpad gestures

3. Previous Research: Tools for supporting designers (developers) to create gestures with ease MAGIC: [Ashbrook et al. 2010] Proton++: [Kin et al. 2012] Gesture Coder: [Lü et al. 2012]

4. Previous Research: 2) Methods for creating a gesture set that are intuitive and guessable bya wide range of users [Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011] A figure from [Wobbrock et al. 2009]

5. Previous research: (2) Methods for creating a gesture set that are intuitive and guessable bya wide range of users Our focus: Supporting end-users Personal gestures for a single user [Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011] A figure from [Wobbrock et al. 2009]

6. Previous research: (2) Methods for creating a gesture set that are intuitive and guessable bya wide range of users Our focus: Supporting end-users Personal gestures for a single user Why? [Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011] A figure from [Wobbrock et al. 2009]

7. Potential Advantages of Self-defined Gestures… Memorability Efficiency Accessibility

8. Memorability Self-defined gestures improve memorability over predefined gestures [Nacentaet al. 2013]

9. Efficiency Gestural shortcuts can be used as an efficient mean of accessing information [Ouyanget al. 2012]

10. Accessibility Customized gestures mayimprove accessibility for people with physical disabilities [Anthony et al. 2013]

11. Our Goal: To investigate the feasibility of end-user gesture creation

12. Our Goal: To investigate the feasibility of end-user gesture creation How do typical users create gestures? What are the challenges therein? How can we support the process?

13. Study With Three Tasks v Task 2 Task 1 Task 2 Task 3 Saliency of Gesture Features Action-Specific Gesture Creation Open-Ended Gesture Creation

14. Study Method Controlled lab study - 20 participants (age from 20 to 35, M=29.3) - Prior experience with touchscreen devices - Single one-hour session with 3 tasks - Think-aloud protocol Apparatus - Samsung Galaxy Tab 2 (10.1’’ running Android 4.0.4)

15. Are users able to create new gestures easily? If not, what are the barriers? Q. v Task 1 Task 2 Task 2 Task 3 Open-Ended Gesture Creation Saliency of Gesture Features Action-Specific Gesture Creation

16. Task 1: Open-ended Gesture Creation “Create as many gestures as possible”

17. Task 1: Open-ended Gesture Creation “Create as many gestures as possible” • For any purpose • Any number of strokes, fingers, hands • As long as they are: • easy to draw, easy to remember, distinguishable

18. Task 1: Open-ended Gesture Creation “Create as many gestures as possible”

19. Task 1: Open-ended Gesture Creation GesturesCreated 12.2 gestures created on average (SD = 8.1, range of 5 to 36) Total number of gestures and the number of arbitrary gestures are correlated (Pearson’s r=.47, p=.037) p3 p5

20. Task 1: Open-ended Gesture Creation GesturesCreated 12.2 gestures created on average (SD = 8.1, range of 5 to 36) Total number of gestures and the number of arbitrary gestures are correlated (Pearson’s r=.47, p=.037) p3 p5 p5

21. Task 1: Open-ended Gesture Creation Difficulties Creating Gestures Tendency to focus on the familiar “I just thought of gestures my tablet PC had.” (P1) “These gestures are all I use, I cannot be more creative” (P8) Opaque nature of gesture recognizer “Can I use all fingers?” (P2)

22. Users felt difficulty in creating new gestures Better understanding of recognizer is needed A. v Task 1 Task 2 Task 2 Task 3 Open-Ended Gesture Creation Saliency of Gesture Features Action-Specific Gesture Creation

23. Q. What is a “good gesture” to end-users? How is it different from recognizer’s perspective? Task 2 Task 2 Task 1 Task 3 Action-Specific Gesture Creation Saliency of Gesture Features Open-Ended Gesture Creation

24. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action 12 Specific Actions Zoom-in Zoom-out Rotate Copy Cut Paste Select-single Select-multiple Previous Next Call-Mom Launch a web-browser

25. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action 12 Specific Actions Zoom-in Zoom-out Rotate Copy Cut Paste Select-single Select-multiple Previous Next Call-Mom Launch a web-browser

26. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Composecustom set of gestures, one per action

27. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Composecustom set of gestures, one per action

28. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Composecustom set of gestures, one per action

29. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Composecustom set of gestures, one per action Create training examples (4 per selected gesture)

30. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Composecustom set of gestures, one per action Create training examples (4 per selected gesture) Rate satisfaction with the custom gesture set

31. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Composecustom set of gestures, one per action Initial example Test recognition accuracy with $N recognizer Create training examples (4 per selected gesture) Training examples Rate satisfaction with the custom gesture set

32. Task 2: Action-Specific Gesture Creation Reasons for selecting a gesture for custom set Others reasons: Generally preferred, fast, consistent, easy to remember, etc.

33. Task 2: Action-Specific Gesture Creation • Only two participants were fully satisfied • ( M=5.3, SD = 1.1 where 1=negative, 7=positive) Need for improvement Inability to improve gesture sets • Participants gave up the opportunity to edit their gesture set to make improvements

34. Task 2: Action-Specific Gesture Creation Low Recognition Potential of the Custom Sets $N recognizer (default setting) with 5-fold cross validation 76–88% accuracydepending on amount of training

35. Customized set can be improved for both user’s and recognizer’s perspective A. Task 2 Task 2 Task 1 Task 3 Action-Specific Gesture Creation Saliency of Gesture Features Open-Ended Gesture Creation

36. What features do users rely on to distinguish between gestures? Q. Task 3 Task 2 Task 1 Task 2 Saliency of Gesture Features Action-Specific Gesture Creation Open-Ended Gesture Creation

37. Task 3: Saliency of Gesture Features Gesture Features Judged 6 features from Rubine’s recognizer [Rubine. 1991] Orientation Scale Aspect Ratio Pattern Repetition Curviness Speed Very slow Very fast slow fast moderate

38. Task 3: Saliency of Gesture Features Gesture Features Judged 6 features from Rubine’s recognizer [Rubine. 1991] 3 touchscreen features Finger Count Orientation Scale Stroke Count Aspect Ratio Pattern Repetition Stroke Order Curviness Speed Very slow Very fast slow fast moderate

39. Task 3: Saliency of Gesture Features “Rank the distinguishability of 9 features” 6 features from Rubine’s recognizer [Rubine. 1991] 3 touchscreen features Finger Count Orientation Scale Stroke Count Aspect Ratio Pattern Repetition Stroke Order Curviness Speed Very slow Very fast slow fast moderate

40. Task 3: Saliency of Gesture Features Objective features are more distinguishable Features that can be consistently interpreted/manipulated are considered distinguishable “Even if the same person is performing the gesture, it might not have the same speed and size” (P7) Pattern Repetition Stroke Order Finger count Aspect Ratio Stroke count Orientation Curviness Speed Scale Very fast More distinctive

41. Task 3: Saliency of Gesture Features Objective features are more distinguishable Features that can be consistently interpreted/manipulated are considered distinguishable “Even if the same person is performing the gesture, it might not have the same speed and size” (P7) Pattern Repetition Stroke Order Finger count Aspect Ratio Stroke count Orientation Curviness Speed Scale Very fast More distinctive

42. Task 3: Saliency of Gesture Features Objective features are more distinguishable Features that can be consistently interpreted/manipulated are considered distinguishable “Even if the same person is performing the gesture, it might not have the same speed and size” (P7) Pattern Repetition Stroke Order Finger count Aspect Ratio Stroke count Orientation Curviness Speed Scale Very fast More distinctive

43. Number of fingers/strokes, stroke order are distinguishable than speed or size A. Task 3 Task 2 Task 1 Task 2 Saliency of Gesture Features Action-Specific Gesture Creation Open-Ended Gesture Creation

44. Summary • Creating new gestures is hard for end-users • Tendency to focus on the familiar • Opaque nature of gesture recognizer • Quality of gesture sets can be improved • Users are not fully satisfied with their gesture sets • Low recognition potential • Objective features are more distinguishable • Finger/stroke count, stroke order are more distinguishable than speed and scale

45. Take-away Message Potential Benefits of Allowing End-User Customization Systematic Support is Needed for End-User Customization Memorability Efficiency Accessibility

46. Future Work Mixed-initiative support for customization Feedback Gesture set System User Train Edits

47. Questions? The Challenges and Potential of End-User Gesture Customization Thank you for listening UranOh1 and Leah Findlater2 1 Department of Computer Science 2 College of Information Studies University of Maryland, College Park uranoh@cs.umd.edu | leahkf@umd.edu 