AAM based Face Tracking with Temporal Matching and Face Segmentation

1. AAM based Face Tracking with Temporal Matching and Face Segmentation Dalong Du

2. Outline • Author Introduction • AAM Introduction • Abstract • Method and Theory • Experiment

3. Author Introduction • Mingcai Zhou • Institute of Automation Chinese Academy of Sciences • Lin Liang • Microsoft Research Asia

4. Author Introduction • Jian Sun • Microsoft Research Asia • joined in July, 2003. • Educational background • BS degree, MS degree and Ph.D degree from Xian Jiaotong University in 1997, 2000 and 2003 • Current research interests • Interactive compute vision (user interface + vision) • Internet compute vision (large image collection + vision) • stereo matching and computational photography

5. Author Introduction • Yangsheng Wang • Director of Digital Interactive Media Lab, Institute of Automation Chinese Academy of Sciences • Educational background • BS degree, MS degree and Ph.D degree from  Huazhong University of Science and Technology

6. AAM Introduction • Shape Model • Appearance (Texture) Model • AAM Model Search

7. AAM—Shape Model • Face Q consists of N landmark points • The geometry information of Q decouples into two parts: • A shape S • Shape is the geometric information invariant to a particular class of transformations • e.g.Or other linear or nonlinear methods • A transformation • θ • e.g. similarity s, R, t Or Affine or others. • Similarity x = (x1,y1, … , xn, yn)T b θ Same shape Different shape

8. AAM—Shape Model • Shape Model Building • Given a set of shapes • Align shapes into common frame • Procrustes analysis • Estimate shape distribution p(x) • Use PCA The aligned shapes

9. AAM—Shape Model • Shape Model Building, continued • Given aligned shapes, { } • Apply PCA • Compute mean and eigenvectors of covar. • P – First t eigenvectors of covar. matrix • b – Shape model parameters

10. AAM—Texture Model • Building Texture Models • For each example, extract texture vector • Normalise vectors (as for eigenfaces) • Build eigen-model Warp to mean shape Texture, g

11. AAM—Texture Model • Warp method

12. AAM—Texture Model • Warp method, continued

13. AAM—Model Search • Find the optimal shape parameters and appearance parameters to minimize the difference between the warped-back appearance and synthesized appearance Computed by the inverse Compositional parameter Update technique map every pixel x in the model coordinate to its corresponding image point

14. Abstract • Problems • Generalization problem • images with cluttered background • How to do? • A temporal matching constraint in AAM fitting • Enforce an inter-frame local appearance constraint between frames • Introduce color-based face segmentation as a soft constraint

15. Method and Theory • Extend basic AAM to Multi-band AAM • The texture(appearance) is a concatenation of three texture band values • The intensity (b) • X-direction gradient strength (c) • Y-direction gradient strength (d)

16. Method and Theory • Temporal Matching Constraint • Select feature points with salient local appearances at previous frame • Optimize the shape parameters to match the local appearances at current frame

17. Method and Theory • Temporal Matching Constraint, continued • : a set of feature points • Selected by a corner detector and some semantic points • : the face appearance of frame t-1 • : the local patch corresponding to the j-th feature point • : the average intensity of j-th patches of frame t-1 and t respectively Normalize the illuminations of two patches

18. Method and Theory • Temporal Matching Constraint, continued • Add a new term to the AAM cost function • Empirically, Can be efficiently minimized based on inverse compositional algorithm

19. Method and Theory • Temporal Matching Constraint, continued • Be resistant to global illumination changes • Match local patches • Do not suffer from the mismatched points • Feature matching is continuously refined by updating the shape parameters during AAM fitting

20. Method and Theory • Initialize shape • Good initial parameters -> good AAM fitting • Method • Selected feature points at frame t-1 • Matched feature points at frame t • Remaining feature points after main direction filter

21. Method and Theory • Initialize shape, continued • M matched points • Estimate the initial shape parameters • represents the consistency of feature points I’s direction • is the estimated position of the point I given the shape parameters p • are the vertex coordinate of the triangle • are the triangle coordinate Gauss-Newton algorithm

22. Method and Theory • Face Segmentation Constrained AAM • Problem: AAM tends to fit the face outline to the background edges • Method: segment the face region using an adaptive color model and constrain AAM fitting

23. Method and Theory • Formalization • Where are the locations of the selected outline points in the model coordinate Wc = 0.01

24. Experiments • RI: robust initialization • TO: temporal matching constraint • FS: face segmentation

25. Experiments

26. Thank you