1 / 39

Ziyong Feng, Shaojie Xu, Xin Zhang , Lianwen Jin, Zhichao Ye, and Weixin Yang

Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System. Ziyong Feng, Shaojie Xu, Xin Zhang , Lianwen Jin, Zhichao Ye, and Weixin Yang. Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012.

avedis
Download Presentation

Ziyong Feng, Shaojie Xu, Xin Zhang , Lianwen Jin, Zhichao Ye, and Weixin Yang

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System ZiyongFeng, Shaojie Xu, Xin Zhang,Lianwen Jin, ZhichaoYe, and WeixinYang Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012

  2. Outline • Introduction • Related Work • Proposed Method • Experimental Results • Conclusion

  3. Introduction

  4. Introduction • Fingertip detectiontakes a very important role of the natural HCI • Challenge : • Variety of hand poses • Occlusion • In this paper: • Propose a real-time finger writing character recognition system using depth information • Accurate and fast (Human Computer Interaction)

  5. Related Work

  6. Related work • Template matching[3]: • Curvature Fitting[6]: [3] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of Circuits, Systems, and Computers (JCSC), 16(3):421–436, 2007. [6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.

  7. ProposedMethod

  8. Flow Chart • Hand Segmentation • Data Conversion • Region Clustering • Fingertip Identification

  9. Hand Segmentation • Extract human body from background: • User ID map ( by Open Natural Interaction (OpenNI) ) • User Generator

  10. Hand Segmentation • Two kinds hand-torso relationship: • 1) Hand is holding up front. • 2) Hand is close to the body. Depth Histogram

  11. Hand Segmentation • Characterize the depth-histogram by two models: • 1) Two component Gaussian mixture model . • 2) Single Gaussian model. • Hand pixels : • Belong to the Gaussian component with smaller mean Two-Component •  :weight of k-th component :maenof k-th component :variance of k-th component d :depth value Expectation-maximization algorithm

  12. Hand Segmentation • One Gaussian fitting: • When the means of two Gaussian are too near • Distribution: • Hand pixels: • Compared with torso, hand takes a few room. • Lower part of p : One-Component • 

  13. Data Conversion • Convert to real world coordinate: • The accuracy of world coordinate is about 1mm. • The following discussions are all based on real-world coordinate. :projected point coordinate d :depth value :camera’s focal length at axis x and y x : real word coordinate

  14. Region Clustering • Clustering algorithm : K-means • Finger part vs. non-finger part (K=2) • Minimize distortion measure J: n-th sample would be assigned to k-th cluster maen of the k-th cluster

  15. Fingertip Identification • After clustering → hand-related region is separated into two parts. • The fingertip: • The farthest point from one cluster to the center of the other cluster ‧Arm point: -the mean of points that have the same maximum depth ‧The fingertip: O X

  16. ExperimentalResults

  17. Experimental Results • Resolution : 480 640 • 30 ftps using OpenNI (KINECT) • Dataset: • 2 subjects • 6 categories • Total 8185 frames

  18. Experimental Results

  19. Experimental Results Near mode (1m) Far mode (1.5m)

  20. Experimental Results • The distribution of errors from a sequence: ‧Fast movement ‧Finger is orthogonal to the camera plane.

  21. Experimental Results • Smoothed trajectory: Mean filter • 90% recognition rate on English characters • 80% on Chinese characters

  22. Conclusion

  23. Conclusion • Proposes a novel real-time fingertip detection and tracking. • Using depth sequences • Accurate and fast on fingertip detection & character recgonition

  24. Real-time Hand Tracking on Depth Images Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-PaoTsai, and Shawmin Lei Visual Communications and Image Processing (VCIP), 2011 IEEE

  25. Outline • Introduction • Proposed Method • Experimental Results • Conclusion

  26. Introduction

  27. Introduction • Most previous works tracked the hand position on color images and relied heavily on skin colorinformation. • Vulnerable tolighting variations and skin color • In this paper: • Propose a hand tracking algorithm that uses depth images only • Real-time and accurate • Hand click detection method

  28. ProposedMethod

  29. Hand Position Detection • Predict the new hand position based on the hand moving velocity: • H : hand moving velocity (estimated from hand positions tracked in previous frames)

  30. Hand Region Segmentation • Hand region: • Connected component in the 3D point cloud P (from 2D depth image) • Seed Point: • d(.,.) : Euclidean distance • The nearest point in the point cloud P from the predicted hand position ‧Seed Point ‧Predicted hand position

  31. Hand Region Segmentation • Connectivity: • Entire hand region: • Using standard region growing techniques • Hand region grows incrementally and stops when: • 1) Two neighboring points are no longer connected • 2) The geodesic distance to the seed point < 30mm Seed Point 250mm

  32. Hand Region Segmentation • A) Rough hand center: • --The point with maximum boundary points in its neighborhood • -- There should be more boundary points around the palm. • B) Refined hand center: (12mm) Mean-Shift (One iteration)

  33. Hand Region Segmentation • C) Hand center after Mean-Shift:

  34. ExperimentalResults

  35. Experimental Results • Resolution : 320 240 • 3GHz Intel Core 2 Duo E8400 • Computational complexity:

  36. Experimental Results

  37. Experimental Results • Ground truth vs. tracked position (in millimeters)

  38. Conclusion

  39. Conclusion • Proposes a real-time hand tracking algorithm on depth images. • Using: • Region Growing • Geodesic distance • Mean-shift • Can be further extended to two-hand tracking:

More Related