Coding Scheme in Gestures Analysis

1. Coding Scheme in Gestures Analysis Liang Zhou Dr. Manolya Kwa

2. What are Gestures and G-Analysis? • Gesture is the special motion(s) which express specific meaning from thought. ( Another form of Speech) • G-Analysis is the studying of human behaviour which purpose to understand the human actions and their thought.

3. Why G-Analysis? • Gesture analysis by computer becomes an important role in science studying field. (Sign language) • it is key role in human and computer interaction research • It has significant scientific value.

4. Why coding gesture? • Coding Gestures is the process of translating attributes or properties of the gestures (i.e Finger Form) into a form that researchers can systematically analyse. • Coding gesture is the important to • HCI need coding gesture. (One of the most important concepts is that of direct manipulation)

5. Problem In Cognitive studying, the abstract concept of coding scheme can’t be implement in HCI. Reason: • Abstract Code • Ambiguous Expression.

6. Cognitive Actions • Physical Actions: L-Action, M-Action, D-Action • Perceptual Action • Functional Action • Conceptual Action: Goals etc.

7. Cognitive Progress

8. Why Coding Gesture in CA? • 1. Speech and Gesture are the same system. • 2. Interpret the thought more clearly than the sentence and words (Better Understanding) • 3. Gesture express the thought instantly but speech systematically . • 4. Gesture Recognition Tools are mature and widely use in Gesture Analysis.

9. Hypothesis

10. Challenge • How to prove it? • How to Code the gesture ? • How to improve the reliability of Coding Scheme? • How to get the sufficient data ?

11. Method • 4 participants : 2 task • Task 1: Cognitive Progress • Task 2: Gesture Capture Phase. • Cognitive progress : Sketching the chair and record the whole phase. • Capture the gesture and motion by 2 cameras. One for body motion, one for Hand motion.

12. Experiment

13. Chair Object

14. Data Collection • Sketching Record 2. Gesture Image.

15. Coding Scheme • Motion:: • attributes: M_Name, Hand_type frequency(default) TimeStamp Position Direction FingerForm • M_Name: {} • Hand_type: {LH,RH,TH} • frequency: {Single, Repeated } • TimeStamp: {ST,FT} • Position: {Left, Lower-Left, Upper-Left, Center, Right, Lower-Right, Upper-Right, Lower} • Direction {(1,0), (1,1),(-1,0),….}

16. Example

17. Coding Scheme • Gesture :: • Attributes : Name, TimeGap, PosSS, Objects • Name: (GestureName) • TimeGap: (Time Second) • PosSS: (Distance) • Objects( parts of chair)

18. Tools • The Anvil could compute statistics on an annotated video sequence and the annotation element which we defined in the coding scheme will be represented the same behavior and grouped together. Then, we will understand the label between Gestures and Objects. Basic on the Kawali‘s experiment result, we will understand that what kind of the gestures is mapping to which behaviors and the gestures is mapping to the goal /goals or not.

19. Current Works • We have found out 6 common gestures • However, because of lack of the data , we just can simply to capture and represent it by our coding scheme. • But we’ve already find that some data in cognitive actions show a strong correlation with gestures.

20. What we find • One of the most occurring gesture: • “This part of the chair I think is too small or big.” • The hand focus on the size of the part they drew. The level of the size and amount should be expressed.. • P1, P3, P4, P5

21. Conclusion • We provide a new methodology to understand the cognitive actions and provide its possibility to implement HCI in design field • Future work : • 1. Coding scheme need to cover more other’s field. For example , facial and Head movement. • 2. More task in the experiment will gain sufficient data to analysis and explore the gesture.

22. Thanks Question?