gesture recognition using salience detection and concatenated hmms
Download
Skip this Video
Download Presentation
Gesture recognition using salience detection and concatenated HMMs

Loading in 2 Seconds...

play fullscreen
1 / 23

Gesture recognition using salience detection and concatenated HMMs - PowerPoint PPT Presentation


  • 93 Views
  • Uploaded on

Gesture recognition using salience detection and concatenated HMMs. Ying Yin [email protected] Randall Davis [email protected] Massachusetts Institute of Technology. System overview. Feature vector sequence. Depth & RGB images. Hand tracking. Hand movement segmentation. Xsens

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Gesture recognition using salience detection and concatenated HMMs' - shiloh


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
gesture recognition using salience detection and concatenated hmms

Gesture recognition using salience detection and concatenated HMMs

Ying Yin

[email protected]

Randall Davis

[email protected]

Massachusetts Institute of Technology

system overview
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

system overview1
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

hand tracking
Hand tracking
  • Kinect skeleton tracking is less accurate when the hands are close to the body or move fast
  • We use both RGB and depth information
    • Skin
    • Gesture salience (motion and closeness to the observer)
input to recognizer
Input to recognizer
  • Feature vector xt
  • From the Kinect data and hand tracking
    • Relative position of the gesturing hand with respect to shoulder center in world coordinate (R3)
  • From the Xsens unit on the hand
    • Linear acceleration (R3)
    • Angular velocity (R3)
    • Euler orientation (yaw, pitch, roll) (R3)
system overview2
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

hand movement segmentation
Hand movement segmentation
  • Part of gesture spotting
  • Train Gaussian models for rest and non-rest positions
  • During recognition, an observation xt is first classified as a rest or a non-rest position
  • It is a non-rest position if
system overview3
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

continuous gesture models
Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End

continuous gesture models1
Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End

continuous gesture models2
Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End

bakis model for nucleus phase
Bakis model for nucleus phase
  • 6 hidden states per nucleus phase in the final model
  • Emission probability: mixture of Gaussians with 6 mixtures

s1

s2

s3

s4

s5

s6

start

p(END|s6)

p(s1)

concatenated hmms
Concatenated HMMs
  • Train an HMM for each phase for each gesture
  • Model termination probability for each hidden state sas p(END|s)
  • EM parameter estimation
concatenated hmms1
Concatenated HMMs
  • After training, concatenate HMMs for each phase to form one HMM for each gesture
  • Compute transition probability from the previous phase to the next phase
  • Ensure
gesture spotting recognition
Gesture spotting & recognition

no nucleus phase

  • Detect rest vs non-rest segments
  • Find concatenated HMM that gives the highest probability
  • Find most probable hidden state sequence using Viterbi
    • Assign hidden states to corresponding phases
    • Identify segment without nucleus phase
gesture recognition result
Gesture recognition result
  • 10 users and 10 gestures and 3 rest positions
  • 3-fold average
gesture recognition result1
Gesture recognition result
  • User independent training and testing
  • 3-fold average
contributions
Contributions
  • Employed novel gesture phase differentiation using concatenated HMMs
  • Used hidden states to
    • identify movements with no nucleus phases
    • accurately detect start and end of nucleus phases
  • Improved hand tracking when the hand is close to the body or moving fast by gesture salience detection
ad