1 / 15

Gesture Recognition

Oleksiy Busaryev John Doolittle. Gesture Recognition. CSE 634 –Autumn 2010. Idea Overview. Extract hand image from webcam Background subtraction, HSV Recognize hand gestures in real-time Statistical matching with Hu moments Local feature classifiers Map to keyboard and mouse

kailey
Download Presentation

Gesture Recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Oleksiy Busaryev John Doolittle Gesture Recognition CSE 634 –Autumn 2010

  2. Idea Overview • Extract hand image from webcam • Background subtraction, HSV • Recognize hand gestures in real-time • Statistical matching with Hu moments • Local feature classifiers • Map to keyboard and mouse • Use in real applications • The Open Racing Car Simulator (TORCS), Google Earth • Platform: C++, OpenCV

  3. Background Subtraction • Calculate mean and variance across first 50 frames • Find Mahalanobis distance between sequence and each new frame • Turn on pixel if distance > threshold • Problem: shadows

  4. HSV-Based Flesh Extraction • Gaussian smoothing to remove color noise • Convert to HSV, threshold hue (90..140) • Combine with initial image • Hue is not enough  • Threshold saturation Captured HSV Hue 90 < Hue < 140 Final

  5. Local Feature Classifier • Find hand contour • Determine and threshold curvature extrema • Positive angle = peak • Negative angle = valley • Average peaks/valleys across 5 frames to smooth out noise • Classify gesture based on number of peaks and valleys

  6. LFC Gestures and Mapping Simulate Mouse Cursor movement based on centroid position Peaks > 7, Valleys = 0 Peaks > 35, Valleys > 30 Ratio of Peaks/Valleys between 1 and 3 (checked last) Peaks + Valleys < 2 |Peaks – Valleys| < 2

  7. Classifier Evaluation • Difficult to threshold extrema • Noise caused points to jump each frame • High threshold could eliminate important points • Settled on taking more than one per finger • Recognition was accurate at first • Increasingly difficult with each new gesture • Sometimes changes rapidly between gesture recognized • Shift in angle of hand can change recognized gesture • Translation/scale/rotation invariant • Achieved real-time performance

  8. Statistical Matching • Compute statistical models of hand gestures • Use Hu moments (rotation & scale invariant) • Match by comparing Mahalanobis distances 100 training images per gesture Gesture recognition example Problem!

  9. Finger Tip Extraction • Estimate curvatures on a polygon – bad  • Use central differences (or higher order formula) • Thresholding angles – good enough  • Compute angle Pi-d Pi Pi+d (d=8) on a polygon Curvature estimation Angle Thresholding

  10. Finger Line Fitting • Bisector of PxPi Py • Px, Py are farthest • points within R Bisector of Pi-d Pi Pi+d Bad  Least squares Bad 

  11. Application – Google Earth

  12. Application: Racing Game Acceleration Brake Left Right

  13. Lessons • Hand image extraction • BS – better with a scene lit so as to eliminate shadow • HSV – better in real life, when lighting conditions are poor • Matching • Feature classifier – better for distinguishing small differences with fingers (e.g. together vs. apart) • Hu moments – more reliable general rough estimation • Analyzing hand contour • Exact curvatures are not reliable enough • Angles, least squares: better for precise tracking

  14. Future Work • Try hystogram-based HSV image extraction • Improve peak/valleys (see convexity defects) • Experiment with large Hu moment databases • Try to deal with partially ocluded gestures • Use eigenvectors to fit line into a finger • Support double click etc. • Make cursor control more intuitive

  15. Work Allocation • OleksiyBusaryev • HSV extraction • Hu-based matching • Finger tip extraction and line fitting • Keyboard simulation, TORCS • John Doolittle • Background subtraction • Local feature classifier • Mouse simulation, Google Earth

More Related