Gesture recognition using salience detection and concatenated HMMs

1. Gesture recognition using salience detection and concatenated HMMs Ying Yin yingyin@csail.mit.edu Randall Davis davis@csail.mit.edu Massachusetts Institute of Technology

2. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

3. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

4. Hand tracking • Kinect skeleton tracking is less accurate when the hands are close to the body or move fast • We use both RGB and depth information • Skin • Gesture salience (motion and closeness to the observer)

5. Hand tracking

6. Hand tracking

7. Input to recognizer • Feature vector xt • From the Kinect data and hand tracking • Relative position of the gesturing hand with respect to shoulder center in world coordinate (R3) • From the Xsens unit on the hand • Linear acceleration (R3) • Angular velocity (R3) • Euler orientation (yaw, pitch, roll) (R3)

8. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

9. Hand movement segmentation • Part of gesture spotting • Train Gaussian models for rest and non-rest positions • During recognition, an observation xt is first classified as a rest or a non-rest position • It is a non-rest position if

10. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

11. Temporal model of gestures

12. Continuous gesture models Pre- stroke Post-stroke Nucleus Rest End

13. Continuous gesture models Pre- stroke Post-stroke Nucleus Rest End

14. Continuous gesture models Pre- stroke Post-stroke Nucleus Rest End

15. Pre-stroke & post-stroke phases

16. Bakis model for nucleus phase • 6 hidden states per nucleus phase in the final model • Emission probability: mixture of Gaussians with 6 mixtures s1 s2 s3 s4 s5 s6 start p(END|s6) p(s1)

17. Concatenated HMMs • Train an HMM for each phase for each gesture • Model termination probability for each hidden state sas p(END|s) • EM parameter estimation

18. Concatenated HMMs • After training, concatenate HMMs for each phase to form one HMM for each gesture • Compute transition probability from the previous phase to the next phase • Ensure

19. Gesture spotting & recognition no nucleus phase • Detect rest vs non-rest segments • Find concatenated HMM that gives the highest probability • Find most probable hidden state sequence using Viterbi • Assign hidden states to corresponding phases • Identify segment without nucleus phase

20. Gesture recognition result visualization

21. Gesture recognition result • 10 users and 10 gestures and 3 rest positions • 3-fold average

22. Gesture recognition result • User independent training and testing • 3-fold average

23. Contributions • Employed novel gesture phase differentiation using concatenated HMMs • Used hidden states to • identify movements with no nucleus phases • accurately detect start and end of nucleus phases • Improved hand tracking when the hand is close to the body or moving fast by gesture salience detection