3D Face Reconstruction from Video

1. 3D Face Reconstruction from Video Unsang Park and Anil K. Jain Pattern Recognition & Image Processing Lab., Computer Science & Engineering • Experimental Results • 2D to 3D face reconstruction • (a)(b)(c) • (a) A pair of 2D images used for the reconstruction, (b) Reconstructed 3D face models at two novel views, and (c) 3D face models acquired using 3D range sensor. • 2D Face recognition with reconstructed 3D face models • Probe images from video are matched against the 2D face image population obtained from the reconstructed 3D face models. (15 subjects for probe and 100 subjects in gallery) • Two commercial face recognition engines (Identix and Cognitec) are used in a fusion scheme of sum-rule with min-max normalization. Schematic Diagram of Face Recognition utilizing 3D face reconstruction from Video • Motivation • Face images in video show large variations in pose and lighting, Current state-of-the art face recognition engines are not very accurate when the probe and gallery images have different pose and lighting. • Taking multiple snapshots with various pose and lighting is less practical. A single 3D face model can generate face images at various pose and lighting conditions. • Building 3D face model with 3D range sensor is expensive. Hence, a practical method of 3D face modeling technique is required. Texture mapping Reconstructed 3D face model Coarse reconstruction Fine reconstruction (TPS) Generic Model 2D video frames (a)(b) (a) Pose and lighting variations appearing in video. (b) Face recognition performance with and without the variations in probe data. Face recognition performance drastically drops as the gallery data does not contain the variations in the probe data. Generate 2D face population Enrollment stage Recognition stage Enroll in 2D face database • Proposed Solution • We propose a 2D to 3D face reconstruction technique to build 3D face models from two video streams without using the 3D range sensor. • Reconstructed 3D face models are used to generate 2D face image population for the robust face recognition invariant to pose and lighting variation. Match Probe Gallery Identity • Fine Reconstruction • The generic model is fitted to the coarse reconstruction model by Thin plate spline (TPS) process. Let U = {ui | i=1, 2, …,n} be the control points on the generic model and V be the control points on the coarse model. The non-linear deformation F(u) is obtained by • The deformation F(u) is applied to all vertices in the generic model for the fine reconstruction. Coarse Reconstruction • Coarse 3D reconstruction is performed based on a set of corresponding points between two images from the two video streams. The set of corresponding points are manually labeled. • A closed form equation is solved to obtain the 3D coordinates given 2D image coordinates (u,v) and calibration matrix (P) as 2D Face recognition utilizing reconstructed 3D face models and multiple number of probe images (1, 10, 20 and 30). Conclusions and Future work • Fast and realistic 3D face reconstruction from 2D video is developed. • 2D face image population obtained from the reconstructed 3D face improved the face recognition performance. • Automatic feature points extraction method for the coarse reconstruction process needs to be developed. • A fully automated system from the video input to 3D face reconstruction and face recognition will be developed. Generic Face Model Construction • A generic model with about 5000 vertices is constructed from the Morphable model that is built using USF 3D face database.