Outline

1. Outline • Peter N. Belhumeur, Joao P. Hespanha, and David J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, 1997.

2. The Goal • Face recognition that is insensitive to large variations in lighting and facial expressions • Note that lighting variability here includes lighting intensity, direction, and number of light sources Computer Vision

3. The Difficulty • It is difficult because the same person with the same facial expression, and seen from the same viewpoint, can appear dramatically different when light sources illuminate the face from different directions Computer Vision

4. Observation • All of the images of a Lambertian surface, taken from a fixed viewpoint, but under varying illumination, lie in a 3D linear subspace of the high-dimensional image space • Image formulation can be modeled by • For a Lambertian surface, the amount of reflected light does not depend on the viewing direction, but only on the cosine angle between the incidence light ray and the normal of the surface • So, for Lambertian surfaces, we have Computer Vision

5. Observation • Therefore, in the absence of shadowing, given three images of a Lambertian surface from the same viewpoint taken under three known, linearly independent light source directions, the albedo and surface normal can be recovered • One can reconstruct the image of the surface under an arbitrary lighting direction by a linear combination of three original images Computer Vision

6. 3D Linear Space Example Computer Vision

7. 3D Linear Space Example Computer Vision

8. The Problem Statement • Given a set of face images labeled with the person’s identity and un unlabeled set of face images from the same group of people, identify each person in the test images Computer Vision

9. Correlation • Nearest neighbor in the image space • If all the images are normalized to have zero mean and unit variance, it is equivalent of choosing the image in the learning set that best correlates with the test image • Due to the normalization process, the result is independent of light source intensity Computer Vision

10. Correlation • Covariance • Correlation Computer Vision

11. Correlation • Problems with correlation • If the images are gathered under varying light conditions, then the corresponding points in the image space may not be tightly clustered • Computationally, it is expensive to compute correlation between two images • All the images have to be stored, which can require a large amount of storage Computer Vision

12. Eigenfaces • As we discussed last time, we can reduce the computation by dimension reduction using PCA • Suppose we have a set of N images and there are c classes • We define a linear transformation • The total scatter of the training set is given by Computer Vision

13. Eigenfaces • For PCA, it chooses to maximize the total scatter of the transformed feature vectors , which is • Mathematically, we have Computer Vision

14. Eigenfaces • When lighting changes, the total scatter is due to the between-class scatter that is useful for classification and also due to the within-class scatter, which is unwanted for classification purposes • When lighting changes, much of the variation from one image to the next is due to the illumination changes • An ad-hoc way of dealing with this problem is to discard the three most significant principal components, which reduces the variations due to lighting Computer Vision

15. Linear Subspaces • Note that all images of a Lambertian surface under different lighting lie in a 3D linear subspace • For each face, use three or more images taken under different lighting directions to construct a 3D basis for the linear subspace • To perform recognition, we simply compute the distance to a new image to each linear subspace and choose the face corresponding to the shortest distance • If there is no noise or shadowing, it would achieve error free classification under any lighting conditions Computer Vision

16. Fisherfaces • Using Fisher’s linear discriminant to find class-specific linear projections • More formally, we define the between-class scatter • The within-class scatter • Then we choose to maximize the ratio of the determinant of the between-class scatter matrix to the within-class scatter of the projected samples Computer Vision

17. Fisherfaces • That is, Computer Vision

18. Comparison of PCA and FDA Computer Vision

19. Fisherfaces • Singularity problem • The within-class scatter is always singular for face recognition • This problem is overcome by applying PCA first, which can be called PCA/LDA Computer Vision

20. Experimental Results • Variation in lighting Computer Vision

21. Experimental Results Computer Vision

22. Experimental Results Computer Vision

23. Experimental Results Computer Vision

24. Variations in Facial Expression, Eye Wear, and Lighting Computer Vision

25. Variations in Facial Expression, Eye Wear, and Lighting Computer Vision

26. Variations in Facial Expression, Eye Wear, and Lighting Computer Vision

27. Variations in Facial Expression, Eye Wear, and Lighting Computer Vision

28. Variations in Facial Expression, Eye Wear, and Lighting Computer Vision

29. Variations in Facial Expression, Eye Wear, and Lighting Computer Vision

30. Glasses Recognition • Glasses / no glasses recognition Computer Vision