Face Image Retrieval by Shape Manipulation

1. CVPR 2011 Brandon Smith (University of Wisconsin-Madison) Shengqi Zhu (University of Wisconsin-Madison) Li Zhang Face Image Retrieval by Shape Manipulation

2. Outline • Introduction • Related Work • System Overview • Technical Details • Experiments

3. Introduction the user drags the tip of the nose leftward to search for similar faces with leftward pose in a sizable database

4. Introduction • Face Image Retrieval • example based • attribute based • Disadvantage • Shape manipulation based

5. Related Work • Being john malkovich. In ECCV, 2010 • “ Ever wanted to be someone else? Now --- you can.”

6. Related Work • Face poser: Interactive modeling of 3d facial expressions using facial priors. In SIGGRAPH, 2010 • generate a new 3D model from example models

7. Related Work • A generative shape regularization model for robust face alignment. In ECCV, 2008

8. Tensor Algebra Given data matrix X ( 2-dimensional) PCA decomposes X The scalar version of Eq.(1) is The 3-dimensional array

9. the matrix form of tensor product

10. System Overview consists of the following four components • Face database construction • Tensor model training • Tensor coefficient recovery • Search by tensor coefficient comparison

11. System Overview • Face database construction • construct a sizable database of aligned faces that exhibit a wide range of pose and facial expression variation • a novel confidence score to filter out poor alignment results

12. System Overview • Tensor model training • Form a 4-dimensional data tensor X, with each of the four dimensions • vertex • Pose • expression • Identity • a new and simple iterative algorithm to achieve this decomposition in the presence of missing data

13. System Overview • Tensor coefficient recovery • Using the Z andUvert, we associate each aligned face in the databasewith three coefficient vectors, cpose, cexpr, and ciden for pose, expression, and identity • Search by tensor coefficient comparison

14. Technical Details • Tensor Decomposition with Missing Data • Even highly structured datasets like Multi - PIE have missing faces • minimizing error function • separate a data tensor X into two parts • estimate Xmissing, Z, Uvert, Upose, Uexpr, and Uiden

15. The error function • initialize Xmissing • the mean average of Xknown ( pose , expression , and identity ) • Iterate between the following steps to minimize Eq. (9) • Fix Xmissing and optimize Z, Uvert, Upose, Uexpr, and Uiden • Fix all the U’s and optimize Xmissing and Z

16. Tensor Coefficient as Facial Feature Vector • Algorithm • Input : Face shape • Output : three coefficient vectors of the face ( pose, expression, and identity ) • Express face shape vector • Z and Uvert are estimated from

17. Estimate all of the c’s from fknownby a function as follows • the rows in Uvert • the errors by Eq. (9) • Iteratively the Eq.(13) • hold two of them constant and update the remaining one until Eq. (13) decreases by 10^-6

18. Searching for Faces Using Tensor Coefficients • Using cpose,cexpr, and ciden enables a user to search images by shape manipulation • Ambiguity • Example : drag the corner of mouth

19. Face Alignment Confidence Measure • alignment confidence score • dnis the average distance from point n to its neighbors • found by the alignment algorithm [4] • Sn range [ 0 , 1 ] • 1 maximum confidence • 0 no confidence

20. Filter the point confidence scores • g(sn) is a Gaussian filter of the confidencescores • alignment confidence for a face shape • N is the number of landmarks

21. Experiments

22. Constructing a Database of Aligned Faces • Dataset • Public Figures (PubFig) dataset • 50,000 images • Training set • Multi-PIE dataset • 1470face • 5 poses • 6 expressions • we supplied 962 faces • not include faces with both non-frontal pose and non-neutral expression

23. Face Retrieval Performance • users only need to edit one or a few face points for our system to find desired faces

25. THANK YOU !