Gesture recognition using salience detection and concatenated hmms
This presentation is the property of its rightful owner.
Sponsored Links
1 / 23

Gesture recognition using salience detection and concatenated HMMs PowerPoint PPT Presentation


  • 57 Views
  • Uploaded on
  • Presentation posted in: General

Gesture recognition using salience detection and concatenated HMMs. Ying Yin [email protected] Randall Davis [email protected] Massachusetts Institute of Technology. System overview. Feature vector sequence. Depth & RGB images. Hand tracking. Hand movement segmentation. Xsens

Download Presentation

Gesture recognition using salience detection and concatenated HMMs

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Gesture recognition using salience detection and concatenated hmms

Gesture recognition using salience detection and concatenated HMMs

Ying Yin

[email protected]

Randall Davis

[email protected]

Massachusetts Institute of Technology


System overview

System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


System overview1

System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


Hand tracking

Hand tracking

  • Kinect skeleton tracking is less accurate when the hands are close to the body or move fast

  • We use both RGB and depth information

    • Skin

    • Gesture salience (motion and closeness to the observer)


Hand tracking1

Hand tracking


Hand tracking2

Hand tracking


Input to recognizer

Input to recognizer

  • Feature vector xt

  • From the Kinect data and hand tracking

    • Relative position of the gesturing hand with respect to shoulder center in world coordinate (R3)

  • From the Xsens unit on the hand

    • Linear acceleration (R3)

    • Angular velocity (R3)

    • Euler orientation (yaw, pitch, roll) (R3)


System overview2

System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


Hand movement segmentation

Hand movement segmentation

  • Part of gesture spotting

  • Train Gaussian models for rest and non-rest positions

  • During recognition, an observation xt is first classified as a rest or a non-rest position

  • It is a non-rest position if


System overview3

System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


Temporal model of gestures

Temporal model of gestures


Continuous gesture models

Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End


Continuous gesture models1

Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End


Continuous gesture models2

Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End


Pre stroke post stroke phases

Pre-stroke & post-stroke phases


Bakis model for nucleus phase

Bakis model for nucleus phase

  • 6 hidden states per nucleus phase in the final model

  • Emission probability: mixture of Gaussians with 6 mixtures

s1

s2

s3

s4

s5

s6

start

p(END|s6)

p(s1)


Concatenated hmms

Concatenated HMMs

  • Train an HMM for each phase for each gesture

  • Model termination probability for each hidden state sas p(END|s)

  • EM parameter estimation


Concatenated hmms1

Concatenated HMMs

  • After training, concatenate HMMs for each phase to form one HMM for each gesture

  • Compute transition probability from the previous phase to the next phase

  • Ensure


Gesture spotting recognition

Gesture spotting & recognition

no nucleus phase

  • Detect rest vs non-rest segments

  • Find concatenated HMM that gives the highest probability

  • Find most probable hidden state sequence using Viterbi

    • Assign hidden states to corresponding phases

    • Identify segment without nucleus phase


Gesture recognition result visualization

Gesture recognition result visualization


Gesture recognition result

Gesture recognition result

  • 10 users and 10 gestures and 3 rest positions

  • 3-fold average


Gesture recognition result1

Gesture recognition result

  • User independent training and testing

  • 3-fold average


Contributions

Contributions

  • Employed novel gesture phase differentiation using concatenated HMMs

  • Used hidden states to

    • identify movements with no nucleus phases

    • accurately detect start and end of nucleus phases

  • Improved hand tracking when the hand is close to the body or moving fast by gesture salience detection


  • Login