Capturing Facial Details by Space-time Shape-from-shading

1. Capturing Facial Details by Space-time Shape-from-shading Yung-Sheng Lo*, I-Chen Lin*, Wen-Xing Zhang*, Wen-Chih Tai†, Shian-Jun Chiou† CAIG Lab, Dept. of CS, National Chiao Tung University* Chunghwa Picture Tubes, LTD.†

2. Outline • Introduction • Acquisition of facial motion • Space-time shape-from-shading • Experiment and results • Conclusion

3. Introduction • Performance-driven method is one of the most straightforward method for facial animation. • Expression details, e.g. wrinkles, dimples, are key factors but difficult for motion capture. Original captured images Deformation without details

4. Introduction (cont.) • Physics-based simulation and blend shape methods try to mimic the details. • But, the synthesized details are not the exact expressions. Blend shape [Z. Deng et al. 2006] Muscle-based [E. Sifakis et al. 2005]

5. Introduction (cont.) • Our goal is to enhance existing motion capture tech. and capture facial details. • With the captured images and directional lighting, our optimization-basedshape-from-shading (SFS) can estimate details from shading in video. Captured video With facial details

6. The proposed method • Combines the benefits of motion capture and shape-from-shading. • Motion capture and stereo reconstruction • accurate on feature points and general geometry. • Unreliable corresponding matching at textureless regions • Shape-from-shading • don’t need detailed point correspondencefor textureless regions. • Estimate relative undulation. • Sensitive to noise. Motion capture + Space-time shape-from-shading.

7. The proposed method

8. Approximate geometry by Mocap • Tracking by block matching and stereo reconstruction. • Deforming a generic face model by radial-basis functions (RBF).

9. Input image Facial details by SFS • Estimating time-varying details by iterative approximating shape V and reflectance R.

10. Space-time constraints • Only SFS is not enough. • For more reliable detailed motions, we proposed using space-time constraints. Highly sensitive to noise After applying our spatial constraints

11. Spatial constraints • Mostly continuous surface • High spatial coherence • For J Є Neighbor(p) • Neighbor(p) denotes the 8-neighbor pixel set • Wj is an adaptive weight • Kcs is the weight for spatial constraints. noise noise Ztp Ztj Ztj Ztj Ztp Ztp Reduce the noise

12. Temporal constraints • Still flicker • According to biomechanics properties: • A human facial surface should gradually transit between expressions. A video image sequence flicker T2 T2 T0 T0 T1

13. Space-time shape-from-shading • Finally, our objective function becomes spatial constraints temporal constraints Space-time shape-from-shading + + = shading constraints

14. Performance issue • if applied our optimization to the whole face. • DOF is too large • Assigned some small windows. • We preferred areas with more wrinkles and creases. Video image sequence N … D.O.F=N*M (pixels) *i(frames) F3 M Fi F2 F1

15. Experiment • Illumination-controlled (single light source) • Two video streams.(HDV, 1280*720 ,30 fps) • We pasted 25 to 30 markers on human face. {C1} {C2}

16. Facial detailed results and Comparison

17. Result of synthesis • Generic model: • 6078 vertices • 6315 polygons deformation (RBF) subdivision per-pixel normal mapping

18. Result of animation

19. Conclusion & Future work • We propose capturing detailed motion by conventional Mocap and advanced shape-from-shading. • Doesn’t need additional devices, paint pigments, or restrict the wrinkle shape. • With spatial and temporal constraints, our optimal shape-from-shading is more reliable. • Reflectance parameters are also estimated.

20. Conclusion & Future work • In addition to Phong model, we will extend the concept to other reflectance models. • E.g Cook-Torrance BRDF model, BSSRDF, etc) • Currently, SFS is only applied to designated segments. An more efficient SFS for the whole face will make our animation more realistic.

21. Thank for your attention! forehead details between eyebrows smile