toolkits for supporting gestures in applications n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Toolkits for Supporting Gestures in Applications PowerPoint Presentation
Download Presentation
Toolkits for Supporting Gestures in Applications

Loading in 2 Seconds...

play fullscreen
1 / 55
efuru

Toolkits for Supporting Gestures in Applications - PowerPoint PPT Presentation

71 Views
Download Presentation
Toolkits for Supporting Gestures in Applications
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Toolkits for Supporting Gestures in Applications Justin Weisz 05-830 UI Software Nov. 16, 2004

  2. What is a gesture? “A single stroke indicates the operation (move text), the operand (the text to be moved), and additional parameters (the new location of the text).” -- Rubine

  3. What is a gesture? “A single stroke indicates the operation (move text), the operand (the text to be moved), and additional parameters (the new location of the text).” -- Rubine

  4. Editing existing objects Creating new objects Uses of gestures

  5. Issuing commands Back Reload page Menu > Copy Uses of gestures

  6. Applications of gesturing - Lightpens

  7. Applications of gesturing - Tablets

  8. Applications of gesturing - PDAs

  9. Applications of gesturing - Video games PowerGlove in “The Wizard”

  10. Applications of gesturing - Video games Black and White - 2001

  11. Rubine [1991] Gesture Recognizers Automated (in a) Novel Direct Manipulation Architecture

  12. Rubine [1991]

  13. Rubine [1991]

  14. Rubine [1991] For the line gesture: recog = [Seq :[handler mousetool:LineCursor] :[[view createLine] setEndpoint:0 x:<start X> y:<start Y>] ]; manip = [recog setEndpoint:1 x:<current X> y:<current Y>]; done = nil;

  15. Rubine [1991] BUT, how are gestures actually represented and recognized? Assumptions: • Gestures are 2D, single strokes • Start and end of a gesture is clearly defined Representation: set of P sample points position & timestamp, preprocessed to remove jitter

  16. Example features: cos of initial angle length of BB diagonal angle between three pts(?) total angle traversed Rubine [1991] Feature vector extracted from G:

  17. Rubine [1991] BUT... “The aforementioned feature set was empirically determined by the author to work well on a number of different gesture sets” -- Rubine

  18. Rubine [1991] Classification Each gesture class represented by a weight vector To classify gesture G: weight of feature i score bias(?) feature i of gesture G Take the highest score:

  19. Rubine [1991] Training Optimal classifier

  20. Rubine [1991] Rejection > 0.95? ACCEPT Gesture G Pr(G matches i) Classification i REJECT g1 g3 g2 mean(i)

  21. Rubine [1991] Evaluation

  22. Rubine [1991] Evaluation

  23. Rubine [1991] Evaluation

  24. Aside: Agate - Landay, Myers [1993]

  25. gdt - Long et al. [1999]

  26. gdt - Long et al. [1999] Newton and Palm users reported: • Gestures are powerful, efficient and convenient • Want more commands to have gestures • Want to define new gestures • Recognition accuracy is not good enough

  27. gdt - Long et al. [1999] Oh Agate, I will make you beautiful!

  28. gdt - Long et al. [1999]

  29. gdt - Long et al. [1999] Distance matrix

  30. gdt - Long et al. [1999] Classification matrix

  31. gdt - Long et al. [1999] Experiment - Hypotheses • “Participants could use gdt to improve their gesture sets.” • “The tables gdt provided would aid designers.” • “PDA users and non-PDA users would perform differently.”

  32. gdt - Long et al. [1999] Experiment - Procedure (pay no attention to the man behind the curtain...)

  33. gdt - Long et al. [1999] Experiment - Results

  34. “Clustering” d Reverse direction existing gesture classes new class gdt - Long et al. [1999] Experiment - Problems with gdt

  35. “Sloppiness” Gesture overloading Delete gdt - Long et al. [1999] Experiment - Problems with gdt

  36. rect copy gdt - Long et al. [1999] Lessons learned • GDT helpful, but participants averaged a 95.4% recognition rate • Tables too confusing, didn’t help performance (better: “Gesture class A is too similar to gesture class B”) • Should be able to create a test set of gestures and run it against a different gesture class

  37. Break time! Muchas gracias to my officemate for the suggestion. Smiling babies make people happy. BE HAPPY!

  38. Problem: Real problem: it is (still) cumbersome to design a system to perform gesture recognition GT2k - Westeyn et al. [2003]

  39. I’m back! Sensors Microphones Cameras Accelerometers <action> GT2k - Westeyn et al. [2003] GT2k system components Data generator Results interpreter

  40. HMM Transition probs. Symbol output probs. Initial state dist. kth symbol in the alphabet Aside: Hidden Markov Models

  41. Aside: Hidden Markov Models

  42. Aside: Hidden Markov Models • Evaluation problem • Given HMM and O={o1,...,oT}, compute Pr(O|HMM) • Forward algorithm • Decoding problem • Given O, compute most likely state sequence that produced O • Viterbi algorithm • Learning problem • Given O, compute transition probs. to maximize likelihood of observing O • Forward-Backward algorithm (aka. Baum-Welch)

  43. GT2k - Westeyn et al. [2003] Grammars MoveForward = Advance Slow_Down Halt MoveBackward = Reverse Slow_Down Halt command = Attention <MoveForward | MoveBackward>

  44. GT2k - Westeyn et al. [2003] Converting raw sensor data to feature vectors 1 56 Attention 57 175 Advance 176 235 Slow_Down 236 250 Halt

  45. train overfit! test GT2k - Westeyn et al. [2003] Training Training and validation procedure

  46. only during continuous recognition GT2k - Westeyn et al. [2003] Accuracy A = accuracy N = number of examples S = # substitution errors (misclassification) D = # deletion errors (failed to recognize a gesture) I = # insertion errors (system hallucinates a gesture)

  47. GT2k - Westeyn et al. [2003] Applications - Gesture Panel gesture = up | down | left | right | up-left | up-right | down-left | down-right Result: 99.20% accuracy on 251 examples (2 substitution errors)

  48. GT2k - Westeyn et al. [2003] Applications - Prescott blinkprint = person_1 | person_2 | person_3 Result: 89.6% accuracy on 48 examples (5 substitution errors, not good!)

  49. GT2k - Westeyn et al. [2003] Applications - TeleSign word = my | computer | helps | me | talk sentence = ( calibrate word word word word word exit ) Result: 90.48% accuracy on 72 examples

  50. GT2k - Westeyn et al. [2003] Applications - Workshop Activity Recognition gesture = hammer | file | sand | saw | screw | vise | drill | clap | use_drawer | grind Result: 93.33% accuracy on 10 examples per activity