Gesture recognition using salience detection and concatenated hmms
Download
1 / 23

Gesture recognition using salience detection and concatenated HMMs - PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on

Gesture recognition using salience detection and concatenated HMMs. Ying Yin [email protected] Randall Davis [email protected] Massachusetts Institute of Technology. System overview. Feature vector sequence. Depth & RGB images. Hand tracking. Hand movement segmentation. Xsens

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Gesture recognition using salience detection and concatenated HMMs' - shiloh


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Gesture recognition using salience detection and concatenated hmms

Gesture recognition using salience detection and concatenated HMMs

Ying Yin

[email protected]

Randall Davis

[email protected]

Massachusetts Institute of Technology


System overview
System overview concatenated HMMs

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


System overview1
System overview concatenated HMMs

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


Hand tracking
Hand tracking concatenated HMMs

  • Kinect skeleton tracking is less accurate when the hands are close to the body or move fast

  • We use both RGB and depth information

    • Skin

    • Gesture salience (motion and closeness to the observer)


Hand tracking1
Hand tracking concatenated HMMs


Hand tracking2
Hand tracking concatenated HMMs


Input to recognizer
Input to recognizer concatenated HMMs

  • Feature vector xt

  • From the Kinect data and hand tracking

    • Relative position of the gesturing hand with respect to shoulder center in world coordinate (R3)

  • From the Xsens unit on the hand

    • Linear acceleration (R3)

    • Angular velocity (R3)

    • Euler orientation (yaw, pitch, roll) (R3)


System overview2
System overview concatenated HMMs

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


Hand movement segmentation
Hand movement segmentation concatenated HMMs

  • Part of gesture spotting

  • Train Gaussian models for rest and non-rest positions

  • During recognition, an observation xt is first classified as a rest or a non-rest position

  • It is a non-rest position if


System overview3
System overview concatenated HMMs

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition


Temporal model of gestures
Temporal model of gestures concatenated HMMs


Continuous gesture models
Continuous gesture models concatenated HMMs

Pre-

stroke

Post-stroke

Nucleus

Rest

End


Continuous gesture models1
Continuous gesture models concatenated HMMs

Pre-

stroke

Post-stroke

Nucleus

Rest

End


Continuous gesture models2
Continuous gesture models concatenated HMMs

Pre-

stroke

Post-stroke

Nucleus

Rest

End



Bakis model for nucleus phase
Bakis concatenated HMMs model for nucleus phase

  • 6 hidden states per nucleus phase in the final model

  • Emission probability: mixture of Gaussians with 6 mixtures

s1

s2

s3

s4

s5

s6

start

p(END|s6)

p(s1)


Concatenated hmms
Concatenated HMMs concatenated HMMs

  • Train an HMM for each phase for each gesture

  • Model termination probability for each hidden state sas p(END|s)

  • EM parameter estimation


Concatenated hmms1
Concatenated HMMs concatenated HMMs

  • After training, concatenate HMMs for each phase to form one HMM for each gesture

  • Compute transition probability from the previous phase to the next phase

  • Ensure


Gesture spotting recognition
Gesture concatenated HMMsspotting & recognition

no nucleus phase

  • Detect rest vs non-rest segments

  • Find concatenated HMM that gives the highest probability

  • Find most probable hidden state sequence using Viterbi

    • Assign hidden states to corresponding phases

    • Identify segment without nucleus phase



Gesture recognition result
Gesture recognition result concatenated HMMs

  • 10 users and 10 gestures and 3 rest positions

  • 3-fold average


Gesture recognition result1
Gesture recognition result concatenated HMMs

  • User independent training and testing

  • 3-fold average


Contributions
Contributions concatenated HMMs

  • Employed novel gesture phase differentiation using concatenated HMMs

  • Used hidden states to

    • identify movements with no nucleus phases

    • accurately detect start and end of nucleus phases

  • Improved hand tracking when the hand is close to the body or moving fast by gesture salience detection


ad