A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction - PowerPoint PPT Presentation

A fast and robust fingertips tracking algorithm for vision based multi touch interaction
1 / 28

  • Uploaded on
  • Presentation posted in: General

A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction. Qunqun Xie, Guoyuan Liang, Cheng Tang, and Xinyu Wu. 2013 10th IEEE International Conference on Control and Automation (ICCA). Outline. Introduction Related Work Proposed Method

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

A fast and robust fingertips tracking algorithm for vision based multi touch interaction

A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction

Qunqun Xie, Guoyuan Liang,

Cheng Tang, and Xinyu Wu

2013 10th IEEE

International Conference on Control and Automation (ICCA)



  • Introduction

  • Related Work

  • Proposed Method

    • Hand localization

    • Fingertips tracking

    • The Multi-touch system

  • Experimental Results

  • Conclusion





  • Multi-touch technology:

    • Sensor Based

      • Directly receive finger touch as input

      • High cost → limits its application to some extent

    • Computer Vision Based

      • Good scalability as well as good performance

Image: Oka, K, Sato, Y, Koike, H. "Real-time fingertip tracking and gesture recognition", IEEE Computer Graphics and Applications, 2012



  • In this paper:

    • Propose a robust fingertip tracking algorithm:

      • Real-time

      • Stereovision-based 3D multi-touch interaction system

      • Skin / Depth / Geometry structure

Related work

Related Work

Related work1

Related work

L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of Circuits, Systems, and Computers (JCSC), 16(3):421–436, 2007.

  • Geometry properties:

    • Curvature

    • Edge or shape

    • Build a model

  • Image Analysis

    • Template matching

    • Color Segmentation

D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.

Related work2

Related work

  • Palm Center:

  • Fingertip Detection





GSP points






Related work3

Related work

  • [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking in Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012

  • [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011

  • [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012

  • [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013

Proposed method


Hand segmentation

Hand Segmentation

Training data:

  • Skin Color filter

    • YCbCr color space

    • Gaussian Mixture Model

      • Describe the skin-color distribute

      • Single Gaussian Model:

      • Gaussian Mixture Model:

Weight of each Gaussian model:

color vector

Hand segmentation1

Hand Segmentation

  • Skin Color filter

    • : how skin-like the color is

    • Expectation Maximization(EM) algorithm

Hand segmentation2

Hand Segmentation

  • Depth Cue:

    • The points with minimum depth are picked as seeds

    • Region grow algorithm



skin + depth

Hand segmentation3

Hand Segmentation

  • Divide wrist and hand:

    • By a boundary curve [18]

      • Minimum depth

      • Boundary curve

r: row index

c :column index

z(r,c) :depth value


range threshold

(related to palm size)


[18] Z. Mo and U. Neumann, “Real-time hand pose recognition using low-resolution depth images,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2.

Palm region extraction

Palm Region Extraction

  • Observation : Palm is a rectangle-like region

  • Method :Project the hand region in all directions

Palm region extraction1

Palm Region Extraction


Palm center localization

Palm Center Localization






Palm center localization1

Palm Center Localization

  • Palm Center:

    • The point with maximum distance from the closest palm boundary[18].

  • The size of palm R:

palm region

palm boundary

Fingertip localization

Fingertip Localization

  • Fingertip :

    The point with maximum distance to the palm center

    (on the contour of each finger)

  • Candidate set F:

P : contour point

C0: palm center


R:palm size



Fingertip localization1

Fingertip Localization

  • Assign an index to each point in candidate set:

  • Sort candidate set by

  • : index

  • F : candidate set

  • C0: palm center

    • the angle of with negative x-axis

Fingertip localization2

Fingertip Localization

  • Distance between successive points :

  • If > → Start & End point subset

  • Fingertips : maximum distance in each subset

Multi touch system

Multi-touch system

  • TUIO (Tangible User Interface Object)

[24] M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza, “Tuio:A protocol for table-top tangible user interfaces,” in Proc. of the The 6th Intl Workshop on Gesture in Human-Computer Interaction and Simulation, 2005.

Experimental results


Experimental results1

Experimental Results

  • Xeon 3.07Ghz workstation

  • frame rate :20Hz on average(real-time)

  • Modules

    • Fingertip tracking

    • TUIO server

    • TUIO client

[10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674





  • Fast and robust fingertip tracking

  • Without pressuring sensing device & extra marks

  • Hand Segmentation

    • Depth / Skin

  • Fingertip Detection

    • Palm region projection

    • Palm center distance from the boundary

    • Fingertip : assign index (angle)

  • Login