Face Alignment Using Cascaded Boosted Regression Active Shape Models

1. Face Alignment Using Cascaded Boosted Regression Active Shape Models Michael Dixon

2. Faces in computer vision • What problems do people work on? • Detection • Alignment • High-level analysis • Face recognition • Facial expression recognition • Face tracking

3. Face alignment • Given an image of a face and an initial guess, localize key facial features • Approaches • Active Shape Model, 1992 • Boosted Regression ASM, 2007

4. Training data • Given many examples, learn a model 1500 hand-labeled face images

5. The Active Shape Model framework Input image Shape Features

6. The Active Shape Model framework Input image Shape Features

7. Shape model • Given many examples of a shape • Learn a set of constraints on allowable shapes

8. Learning a shape model • Represent as a linear subspace Mean face shape Principal variations from the mean

9. The Active Shape Model framework Input image Shape Features

10. Feature model • Given a patch near a facial feature, predict the correct position of that feature Given Predict

11. Learning a feature model • Generate training examples with known feature positions • Train a regression model to predict the correct displacement

12. Boosted regression • Goal: Learn a function to predict a set of target values • Boosting builds a strong regression model from many weak models • Evaluate a large pool of possible weak regression functions • Select the function with the lowest error and add it to the strong regression model • Update the target values and repeat

13. Weak regression model Haar wavelet response • hm = • The sum of all pixel values under the white box minus the sum of all pixel values under the black box Haar wavelet features Weak regression function

14. Weak regression example a = -0.027 b = 0.012 t = 21.7 displacement displacement hm hm fit weak regression function to data

15. Strong regression model Ground-truth displacement 25 weak regression functions combined into a strong regression function Predicted displacement

16. The Active Shape Model framework • Combining the shape and feature models Alignment Shape Features

17. Fitting using Boosted Regression ASM • Initialize the feature positions • Iteratively • Predict feature positions using regression model • Constrain to fit the shape model • Update feature positions

18. Limitations of the previous work • How often does the boosted regression feature model improve on the initial estimate? Any improvement Some improvement Improved by at least 50% Significant improvement Percent that improved Predicted position vs. actual position Displacement (in pixels)

19. Accuracy trade-off • Regression model can’t accurately predict both large and small displacements Model trained on large displacements Model trained on small displacements Some improvement Some improvement Significant improvement Percent that improved Percent that improved Significant improvement Displacement (in pixels) Displacement (in pixels)

20. Proposed solution • Train multiple models (coarse to fine) and apply them in sequence Fine regression model Coarse regression model Percent that improved Displacement (in pixels)

21. Cascaded Boosted Regression ASM Face Detector Boosted Regression ASM 15 iterations Alignment Image Cascaded Boosted Regression ASM Face Detector Stage 1 5 iterations Stage 2 5 iterations Stage 3 5 iterations Alignment Image

22. Learning an alignment cascade • Train a new stage of the cascade using the output of the previous stage • Use a face detector as the initial stage • For each stage • Measure error distribution of each feature • Generate training examples from the error distribution • Train new feature models • Align all images using the updated model to get a new error distribution

23. Qualitative comparison Boosted Regression ASM Cascaded Boosted Regression ASM

24. Quantitative evaluation • Error metric: where: • di is the distance between the estimated position and the ground truth position of the ith point • s is the inter-ocular distance • An alignment is only as good as its worst point Inter-ocular distance, s Alignment vs. Ground-truth

25. Results • Evaluated on 500 unseen test images 73% Cascaded Standard Average face Cumulative error distribution 19% 3% Alignment error

26. Results • Alignment accuracy after each stage Cumulative error distribution Median alignment error Stage 1 Stage 2 Stage 3 Alignment error Stage

27. Conclusions • Boosted Regression ASMs are a newly proposed method for performing face alignment • Training a cascade of Boosted Regression ASMs can significantly improve alignment accuracy

28. Questions?