Download
oleksiy busaryev john doolittle n.
Skip this Video
Loading SlideShow in 5 Seconds..
Gesture Recognition PowerPoint Presentation
Download Presentation
Gesture Recognition

Gesture Recognition

270 Views Download Presentation
Download Presentation

Gesture Recognition

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Oleksiy Busaryev John Doolittle Gesture Recognition CSE 634 –Autumn 2010

  2. Idea Overview • Extract hand image from webcam • Background subtraction, HSV • Recognize hand gestures in real-time • Statistical matching with Hu moments • Local feature classifiers • Map to keyboard and mouse • Use in real applications • The Open Racing Car Simulator (TORCS), Google Earth • Platform: C++, OpenCV

  3. Background Subtraction • Calculate mean and variance across first 50 frames • Find Mahalanobis distance between sequence and each new frame • Turn on pixel if distance > threshold • Problem: shadows

  4. HSV-Based Flesh Extraction • Gaussian smoothing to remove color noise • Convert to HSV, threshold hue (90..140) • Combine with initial image • Hue is not enough  • Threshold saturation Captured HSV Hue 90 < Hue < 140 Final

  5. Local Feature Classifier • Find hand contour • Determine and threshold curvature extrema • Positive angle = peak • Negative angle = valley • Average peaks/valleys across 5 frames to smooth out noise • Classify gesture based on number of peaks and valleys

  6. LFC Gestures and Mapping Simulate Mouse Cursor movement based on centroid position Peaks > 7, Valleys = 0 Peaks > 35, Valleys > 30 Ratio of Peaks/Valleys between 1 and 3 (checked last) Peaks + Valleys < 2 |Peaks – Valleys| < 2

  7. Classifier Evaluation • Difficult to threshold extrema • Noise caused points to jump each frame • High threshold could eliminate important points • Settled on taking more than one per finger • Recognition was accurate at first • Increasingly difficult with each new gesture • Sometimes changes rapidly between gesture recognized • Shift in angle of hand can change recognized gesture • Translation/scale/rotation invariant • Achieved real-time performance

  8. Statistical Matching • Compute statistical models of hand gestures • Use Hu moments (rotation & scale invariant) • Match by comparing Mahalanobis distances 100 training images per gesture Gesture recognition example Problem!

  9. Finger Tip Extraction • Estimate curvatures on a polygon – bad  • Use central differences (or higher order formula) • Thresholding angles – good enough  • Compute angle Pi-d Pi Pi+d (d=8) on a polygon Curvature estimation Angle Thresholding

  10. Finger Line Fitting • Bisector of PxPi Py • Px, Py are farthest • points within R Bisector of Pi-d Pi Pi+d Bad  Least squares Bad 

  11. Application – Google Earth

  12. Application: Racing Game Acceleration Brake Left Right

  13. Lessons • Hand image extraction • BS – better with a scene lit so as to eliminate shadow • HSV – better in real life, when lighting conditions are poor • Matching • Feature classifier – better for distinguishing small differences with fingers (e.g. together vs. apart) • Hu moments – more reliable general rough estimation • Analyzing hand contour • Exact curvatures are not reliable enough • Angles, least squares: better for precise tracking

  14. Future Work • Try hystogram-based HSV image extraction • Improve peak/valleys (see convexity defects) • Experiment with large Hu moment databases • Try to deal with partially ocluded gestures • Use eigenvectors to fit line into a finger • Support double click etc. • Make cursor control more intuitive

  15. Work Allocation • OleksiyBusaryev • HSV extraction • Hu-based matching • Finger tip extraction and line fitting • Keyboard simulation, TORCS • John Doolittle • Background subtraction • Local feature classifier • Mouse simulation, Google Earth