비전기반 HRI

1. 비전기반 HRI Face Detection & Recognition

2. Face Recognition :the History, Approaches, and Challenges

3. Contents • General FR Process • Face Detection • What is Face Detection? • Why Face Detection is Important? • Why Face Detection Is Difficult? • Research Issues • Procedure of Face Detection • Face Recognition • What is Face Recognition? • Why Face Recognition is Important? • Why Face Recognition Is Difficult? • Research Issues • Procedure of Face Recognition • DB • Conclusion • References

4. General FR Process • Face Tracking • Pattern Recognition • Motion Estimation • Skin Color • etc Face Detection Feature Extraction • Eye Detection • Facial Feature Tracking • Gaze Estimation Face Recognition • Appearance-based (Holistic) • Model-based Example Pictures

5. What is Face Detection • Identify and locate human faces in an image regardless of their • Position • Scale • Orientation • Pose • And illumination • Recent results have demonstrated excellent results. • Fast, multi pose, partial occlusion

6. Why Face Detection is Important? • First step for any fully automatic face recognition system. • First step in many surveillance systems • Face is a highly non-rigid object • Lots of applications • A step towards Automatic Target Recognition(ATR) or generic object detection/recognition

7. Pose Frontal, 45 degree, profile, upside down Presence or absence of structural components Beards, mustaches, and glasses Facial Expression Occlusion Faces may be partially occluded by other objects Imaging Conditions Lighting (spectra, source distribution and intensity) and camera characteristics (sensor response, gain control, lenses), resolution Why Face Detection is Difficult?

8. Research Issues • Representation: How to describe a typical face? • Scale: How to deal with face of different size? • Speed: How to speed up the process? • Post Processing: How to combine detection results? • Target Application Domain: Single image, video • Representation: Holistic, feature, etc • Pre Processing: Histogram equalization, etc • Cues: Color, motion, depth, voice, etc • Classifier Design: Ensemble, cascade • Post Processing: Combing detection results

9. Research Issues • Focus on Detecting • Upright, frontal faces • In a single gray-scale image • With decent resolution • Under good lighting conditions • For detecting faces in low-resolution images

10. Standard Test Sets • MIT test set • (http://www.cs.cmu.edu/~har) • Subsumed by CMU test • CMU test set • (http://www.cs.cmu.edu/~har) • 130 gray scale images with a total of 507 frontal faces • CMU profile face test set • (http://eyes.ius.cs.cmu.edu/usr20/ftp/testing_face_images.tar.gz) • 208 images with faces in profile views • Kodak data set(Eastman Kodak Corp) • Faces of multiple size, pose and varying lighting conditions in color images

11. Procedure of Face Detection • The procedure of face detection could be divided as preprocessing step, face detection step, and post-processing • For preprocessing step, • illumination compensation techniques like histogram equalization, normalization to zero mean and unit variance on the analysis window, and modified census transform have been proposed. • For detecting faces, • many classification algorithms have been proposed to classify the face and nonface patterns such as: skin color based approaches. SVM, gaussian mixture model, maximum likelihood, neural network, and adaboost. • Finally, for post-processing step, • the algorithms usually group detected faces which is located in the similar position. Then, they select only one face from each face group and determine the size, location, and rotation of the selected face.

12. Census Transform • Census Transform • Zabin and Woodfill proposed an illumination insensitive local transform method called census transform(CT) which is an ordered set of comparisons of pixel intensities in a local neighborhood representing which pixels have lesser intensity than the center • Since census transform transforms pixel values by comparison with center pixel value, it can not transforms the pixel values equal to center pixel value.

13. Modified Census Transform • Modified Census Transform • Froba and Ernst proposed modified census transform(MCT) to solve this problem • MCT is an ordered set of comparisons of pixel intensities in a local neighborhood representing which pixels have lesser intensity than the average pixel value. • They could determine all of the 511(29 -1) structure kernels defined on a 3 * 3 neighborhood, while CT has only 256 structure kernels. • Figure shows some examples of MCT patterns and the result image when MCT is applied to images with various illumination change.

14. Adaboost and MCT • Adaboost and MCT • we construct the weak classifier which classifies the face and non-face patterns and the strong classifier which is the linear combination of weak classifiers. The weak classifier consists of the set of feature locations and the confidence values for each MCT pattern.

15. Results • Results • We tested our algorithm on CMU+MIT frontal face test set.

16. What is Face Recognition? • Face recognition scenarios can be classified into two types, (i) face verification (or  authentication) and (ii) face identification (or recognition). • Face verification (”Am I who I say I am?”) is a one-to-one match that compares a query face image against a template face image whose identity is being claimed. • Face identification (”Who am I?”) is a one-to-many matching process that compares a query face image against all the template images in a face database to determine the identity of the query face.

17. Why Face Recognition is Important? • Face recognition has received substantial attention from both research communities and the market, but still remained very challenging in real applications. • Lots of Applications

18. Why Face Recognition is Difficult? • Human face image appearance has potentially very large variations due to • Pose • Illumination (including indoor / outdoor) • Facial expression • Occlusion(e.g., sunglasses) • Facial hair • Aging

19. Why Face Recognition is Difficult?

20. Why Face Recognition is Difficult?

21. Why Face Recognition is Difficult?

22. Why Face Recognition is Difficult?

23. Research Issue • A number of typical algorithms are presented, being categorized into appearance-based and model-based schemes. • Appearance-based methods, three linear subspace analysis schemes are presented, and several non-linear manifold analysis approaches for face recognition are briefly described. • Model-based approaches are introduced, including Elastic Bunch Graph matching, Active Appearance Model and 3D Morphable Model methods.

24. Research Issue • Appearance-based (View-based) Face Recognition • Most of these techniques depend on a representation of images that induces a vector space structure • Model-based FR • The model-based face recognition scheme is aimed at constructing a model of the human face. The model-based scheme usually contains three steps: • 1) Constructing the model • 2) Fitting the model to the given face image • 3) Using the parameters of the fitted model as the feature vector to calculate the similarity between the query face and prototype faces in the database to perform the recognition.

25. Research Issue • A large number of systems has emerged that are capable of achieving recognition rates of greater than 90% under controlled conditions. Successful application under real world conditions remains a challenge though. • Most research has been limited to frontal views obtained under standardized illumination on the same day with absence of occlusion and with neutral facial expression or slight smile.

26. - - = = Extra personal Intra personal Face Recognition using LFA • Face recognition is a multi-class problem, but, we propose to train adaboost based on the intra-personal and extra-personal variation in the LFA feature space. • The thinking of the face recognition method of Moghaddam and Pentland is to convert the multi-class problem to two-class problem.

27. Face Recognition using LFA • A Robust Feature Extraction using LFA • LFA (Local Feature Analysis) is known as a local method for face recognition, because it constructs kernels which detect local structures of a face. • However LFA addressed only image representation, and has problems for recognition. • LFA locally uses the eigenface method for a few single parts of the face(e.g. eyes, nose, mouth) and additionally determines their geometric proportions to each other. • LFA-based face recognition is relatively insensitive with respect to changes in expression and illumination.

28. Databases for FR • FERET database • http://www.itl.nist.gov/iad/humanid/feret/ • XM2TVS database • http://www.ee.surrey.ac.uk/Research/VSSP/xm2vtsdb/ • UT Dallas database • http://www.utdallas.edu/dept/bbs/FACULTY PAGES/otoole/database.htm • Notre Dame database • http://www.nd.edu/∼cvrl/HID-data.html • MIT face databases • ftp://whitechapel.media.mit.edu/pub/images/ • Shimon Edelman’s face database • ftp://ftp.wisdom.weizmann.ac.il/pub/FaceBase/ • CMU face detection database • http:// www.ius.cs.cmu.edu/IUS/dylan usr0/har/faces/test/ • CMU PIE database • http:// www.ri.cmu.edu/projects/project 418.html • Stirling face database • http://pics.psych.stir.ac.uk • M2VTS multimodal database • http://www.tele.ucl.ac.be/M2VTS/

29. Databases for FR • Yale face database • http:// cvc.yale.edu/projects/yalefaces/yalefaces.html • Yale face database B • http:// cvc.yale.edu/projects/yalefacesB/yalefacesB.html • Harvard face database • http:// hrl.harvard.edu/pub/faces • Weizmann face database • http://www.wisdom.weizmann.ac.il/∼yael/ • UMIST face database • http://images.ee.umist.ac.uk/danny/database.html • Purdue University face database(AR DB) • http://rvl1.ecn.purdue.edu/∼aleix/aleix_face_DB.html • Olivetti face database(ORL) • http://www.cam-orl.co.uk/facedatabase.html • Oulu physics-based face database • http://www.ee.oulu.fi/research/imag/color/pbfd.html • Asian Face Image Database PF01 • http://nova.postech.ac.kr/special/imdb/imdb.html • Face Database Information • http://web.mit.edu/emeyers/www/face_databases.html

30. 3D Databases for FR • UND • 275 subjects, 943 scans • Shape + texture • FRGC • 400 subjects, 4007 scans • Shape + texture • 3D_RMA • 120 subject, 6 scans • Shape only • GavabDB • 61 subjects (9 scans) • Shape only • Pose, expression variations • USF Database • 357 scans • 3DFS generator • Custom Face Databases • 12 persons to ~6000 persons (A4Vision) UND USF GavabDB 3DFS

31. Conclusion • Numerous methods have been proposed for face recognition based on image intensities • Many of these methods have been successfully applied to the task of face recognition, but they have advantages and disadvantages • Although many face recognition techniques have been proposed and have shown significant promise, robust face recognition is still difficult • There are at least three major challenges • Illumination • Pose • Recognition in outdoor imagery • Image-based face recognition is still a very challenging topic after decades of exploration!!!

32. References • M. M.-H. Yang, D. J. Kriegman, and N. Ahuja, H. Yang, D. J. Kriegman, and N. Ahuja, “Detecting Faces in Images: A Survey” IEEE Transactions on Pattern Analysis and Machine Intelligence Intelligence (PAMI), vol. 24, no. 1, pp. 34-58, 2002. • Face Recognition: A Literature Survey, 2003 • W. Zhao et. al. • Image Analysis for Face Recognition, 2003 • Xiaoguang Lu • Quo vadis Face Recognition, 2001 • Ralph Gross • Handbook of Face Recognition, 2004 • Stan Z. Li et. al. • AUTOMATED BIOMETRICS, 2000 • David D. Zhang

33. Introduction to Manifold Learning

34. z x: coordinate for z x Terminology (1) • Manifold – Generalized “subspace” in which is locally Euclidean M x2 R2 x1

35. Terminology (2) • Manifold Learning Given a finite sampling of a d-dimensional manifold find an embedding of into a subset without any prior knowledge about d.

36. Outline • Introduction • Linear Methods • Principle Component Analysis • Multidimensional Scaling • Manifold Learning • IsoMap • Locally Linear Embedding

37. Introduction (1) • Motivation • Observe high-dimensional data • Face Image data, Facial expression features, Video sequences, gene data, many more… • Hopefully, discover compact representations of high-dimensional data dimensionality reduction • Why do dimensionality reduction? • More features provide more information, which gives potentially higher accuracy • Unfortunately, more features make harder to train a classifier  the curse of dimensionality • How can we obtain this Low-dimension structure of representing the high-dimension data?

38. Introduction (2) • A view of dimensionality reduction Example) 64x64 = 4096 dimension image • The task of dimensionality reduction is to find a small number of features to represent a large number of observed dimensions.

39. Introduction (3) • Two approaches to discover low dimensional structures for data in high dimension. • Linear Approaches • Principal component analysis. • Multi dimensional scaling. • Non Linear Approaches (Manifold Learning) • ISOMAP • Local Linear Embedding

40. Principal component Principle Component Analysis (1) • PCA finds the best projection that preserves and maximize variance. Based on eigen-decomposition of data covariance matrix

41. Principle Component Analysis (2) • PCA formally finds the orthogonal basis that maximize the variance of

42. Multidimensional Scaling (1) • MDS is the search for a low dimensional space in which points in the space represent the objects and such that the distances between the points in the space, , match, as well as possible, the original dissimilarities zi xi zj dij MDS xj Items in original space Data points in low-D Euclidean space Goal: find xi s that make as “close” to as possible

43. Multidimensional Scaling (2) • Given only the interpoint distances (dissimilarity) . • Step: 1. From , calculate . 2. Spectrally decompose 3. 4. Rank projections Y closest to X is Y = Centering Matrix B’s top rank eigen vectors/eigenvalues Coordinators with Pairwise distances D

44. PCA vs. MDS • In the Euclidean case, MDS only differs from PCA by starting with given D and calculating X . • Starting with D is useful when D is not Euclidean, but a dissimilarity function of the data.

45. Recall the Classical Linear Methods • PCA and Euclidean MDS are simple, efficient, and guaranteed to optimize their criterions. • Because they are linear, they cannot find nonlinear structure in the data. • Example: • Ideally, the data would be represented by the arclength Need a nonlinear process

46. IsoMap (1) • IsoMap Vocabulary: • Isometric :Distance Preserving • Geodesic distance: Shortest curve along the manifold connecting two point • Geodesic distances can be approximated by finding shortest paths in a graph.

47. IsoMap (2) • Recall Classical MDS - Given a set of (all) distance measurements, MDS try to find optimal Euclidean-distance reconstruction • What we really want: • Find distance measurements along manifold (geodesics) • Find low-dim reconstruction which also has these geodesic distances

48. IsoMap [3] • Outline of the Algorithm • Construct neighborhood graph • All points within some fixed radius or K-nearest neighbors • Compute shortest paths (Geodesic distance) • Short Path Problem (in Graph Theory) • Construct d-dimensional embedding • Apply classical MDS

49. IsoMap (4)

50. IsoMap (5)