1 / 28

Model-Based Hand Tracking with Texture, Shading and Self-occlusions

Model-Based Hand Tracking with Texture, Shading and Self-occlusions. CVPR 2008. Martin de La Gorce , Member, IEEE David J. Fleet, Senior, IEEE Nikos Paragios , Senior, IEEE. Model-Based 3D Hand Pose Estimation from Monocular Video. PAMI 2011. 韋弘 2011/03/21. Outline. Introduction

geona
Download Presentation

Model-Based Hand Tracking with Texture, Shading and Self-occlusions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Model-Based Hand Tracking with Texture, Shading and Self-occlusions CVPR 2008 Martin de La Gorce, Member, IEEE David J. Fleet, Senior, IEEE Nikos Paragios, Senior, IEEE Model-Based 3D Hand Pose Estimation from Monocular Video PAMI 2011 韋弘 2011/03/21

  2. Outline • Introduction • Generative Model • Parameter Estimation • Experimental Results • Conclusion

  3. Introduction • Goal • Recover 3D hand pose from monocular video InputImage Synthetic Image (3D Model)

  4. Introduction • Challenges • Search • Hands have about 30 degrees of freedom • Fast accelerations make prediction difficult at 30fps • Depth uncertainty and reflection ambiguities exist • Image measurement • Parts of the hand have similar colors • Surface texture is limited • Edges often ambiguous • Self-occlusions are ubiquitous

  5. Introduction Observed Edges Color /Silhouette • edge and silhouette measurements often leave depth uncertainty and reflection ambiguities unresolved • Shading is an important additional cue

  6. Outline • Introduction • Generative Model • Synthesis • Objective Function • Parameter Estimation • Experimental Results • Conclusion

  7. Synthesis • Geometry- q • Surface:1000 triangular facets • Skeleton:18 bones, 22 DOFs • 54scaling parameters:3 per bone (morphological parameters) • Pose vector q:22+6-dim (global position & orientation w.r.t. camera)

  8. Synthesis • Shading - L • Illuminant L :4-dim vector • Distant point source + ambient light • Lambertian reflectance model • Gouraud shading model

  9. Synthesis • Texture - T • Patch-based texture mapping • Linear equality constraints • Ensure reflectance continuity over the entire surface

  10. Objective Function • Parameters: • Hand pose q, illuminant L and texture T • Synthesized image : • Estimation problem: • Find parameters (q, L, T) that minimize the energy E Image domain Error function - Squared-error

  11. Objective Function • Residual image: • Energy:

  12. Objective Function • Background • median image from several frames for static camera • RGB density function pbg(I) • re-express the residual as: synthetic silhouette interior

  13. Outline • Introduction • Generative Model • Parameter Estimation • Gradient With Respect to Pose and Lighting • Model Registration • Texture Update • Experimental Results • Conclusion

  14. Online Pose Tracking • Initial frame: • Rough manual initialization with canonical pose • Minimize E to find pose, lighting and morphological • texture map equal to mean skin color • For each subsequent frame: • Initialize search by extrapolating previous estimates • Minimize E to refine pose and lighting • Update texture map given pose and lighting Find the gradient of Ew.r.t. pose and lighting

  15. Gradient w.r.t. Pose and Lighting • The gradient of Ew.r.t. lighting L is straightforward • But te derivative of E w.r.tq is not straightforward • is not differentiable w.r.t. q at occlusion boundaries • So integration & differentiation cannot commute 0 1

  16. Gradient w.r.t. Pose and Lighting • 1D illustration • Residual function • Energy function • b is a function of q , when q varies • r0 & r1 varies (e.g. due to shading changes) • the boundary location b moves 0 1

  17. Fundamental theorem of calculus Gradient w.r.t. Pose and Lighting • 1D illustration • Energy derivative(total derivative of E) 0 1

  18. Gradient w.r.t. Pose and Lighting • General 2D case occlusion boundaries boundary velocity boundary normal

  19. Direction: Gradient BFGS Hessian approximation Line search: Update: Model Registration • the model is registered to each new frame • Optimization via Sequential Quadratic Programming (to enforce joint limits)

  20. Model Registration • Newton method • Quasi-Newton method • BFGS update

  21. Texture Update • Texture update by minimization of • Smoothing term between neighboring texels

  22. Outline • Introduction • Generative Model • Parameter Estimation • Experimental Results • Conclusion

  23. Experimental Results • Initialization with canonical hand pose • Optimization to find pose, lighting and morphological parameters First frame Synthetic Image Residual Image

  24. Experimental Results Synthetic Image Input Image Residual Image Synthetic sideview

  25. Stenger’s Data without Pose-Space Reduction Synthetic Image Input Image Residual Image Synthetic sideview

  26. Result with Two Hands

  27. Outline • Introduction • Generative Model • Parameter Estimation • Experimental Results • Conclusion

  28. Conclusion • Improved generative model: • inclusion of pose, illuminant, texture information • an objective function is provided in order to deal with occlusions • Tracking process • Minimize the objective function w.r.t. pose, illuminant • Using a sequential quadratic programming method with adapted BFGS Hessian approximation (combine BFGS quasi-Newton method with the linear joint limit constraints) • Effective optimization: • gradient-based Quasi-Newton energy minimization

More Related