gesture recognition using salience detection and concatenated hmms n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Gesture recognition using salience detection and concatenated HMMs PowerPoint Presentation
Download Presentation
Gesture recognition using salience detection and concatenated HMMs

Loading in 2 Seconds...

play fullscreen
1 / 23

Gesture recognition using salience detection and concatenated HMMs - PowerPoint PPT Presentation


  • 93 Views
  • Uploaded on

Gesture recognition using salience detection and concatenated HMMs. Ying Yin yingyin@csail.mit.edu Randall Davis davis@csail.mit.edu Massachusetts Institute of Technology. System overview. Feature vector sequence. Depth & RGB images. Hand tracking. Hand movement segmentation. Xsens

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Gesture recognition using salience detection and concatenated HMMs' - shiloh


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
gesture recognition using salience detection and concatenated hmms

Gesture recognition using salience detection and concatenated HMMs

Ying Yin

yingyin@csail.mit.edu

Randall Davis

davis@csail.mit.edu

Massachusetts Institute of Technology

system overview
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

system overview1
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

hand tracking
Hand tracking
  • Kinect skeleton tracking is less accurate when the hands are close to the body or move fast
  • We use both RGB and depth information
    • Skin
    • Gesture salience (motion and closeness to the observer)
input to recognizer
Input to recognizer
  • Feature vector xt
  • From the Kinect data and hand tracking
    • Relative position of the gesturing hand with respect to shoulder center in world coordinate (R3)
  • From the Xsens unit on the hand
    • Linear acceleration (R3)
    • Angular velocity (R3)
    • Euler orientation (yaw, pitch, roll) (R3)
system overview2
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

hand movement segmentation
Hand movement segmentation
  • Part of gesture spotting
  • Train Gaussian models for rest and non-rest positions
  • During recognition, an observation xt is first classified as a rest or a non-rest position
  • It is a non-rest position if
system overview3
System overview

Feature vector sequence

Depth & RGB images

Hand tracking

Hand movement segmentation

Xsens

data

Feature vector sequence with

movement

Gesture spotting & recognition

continuous gesture models
Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End

continuous gesture models1
Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End

continuous gesture models2
Continuous gesture models

Pre-

stroke

Post-stroke

Nucleus

Rest

End

bakis model for nucleus phase
Bakis model for nucleus phase
  • 6 hidden states per nucleus phase in the final model
  • Emission probability: mixture of Gaussians with 6 mixtures

s1

s2

s3

s4

s5

s6

start

p(END|s6)

p(s1)

concatenated hmms
Concatenated HMMs
  • Train an HMM for each phase for each gesture
  • Model termination probability for each hidden state sas p(END|s)
  • EM parameter estimation
concatenated hmms1
Concatenated HMMs
  • After training, concatenate HMMs for each phase to form one HMM for each gesture
  • Compute transition probability from the previous phase to the next phase
  • Ensure
gesture spotting recognition
Gesture spotting & recognition

no nucleus phase

  • Detect rest vs non-rest segments
  • Find concatenated HMM that gives the highest probability
  • Find most probable hidden state sequence using Viterbi
    • Assign hidden states to corresponding phases
    • Identify segment without nucleus phase
gesture recognition result
Gesture recognition result
  • 10 users and 10 gestures and 3 rest positions
  • 3-fold average
gesture recognition result1
Gesture recognition result
  • User independent training and testing
  • 3-fold average
contributions
Contributions
  • Employed novel gesture phase differentiation using concatenated HMMs
  • Used hidden states to
    • identify movements with no nucleus phases
    • accurately detect start and end of nucleus phases
  • Improved hand tracking when the hand is close to the body or moving fast by gesture salience detection