Automated recognition of facial expressi ns and identity 2003 UCIT Progress Report

1. Automated recognition of facial expressi ns and identity2003 UCIT Progress Report Ioulia Guizatdinova Research Group for Emotions, Sociality, and Computing University of Tampere 10.01.2004

2. Contents • Research problems • Aims and Tasks • Facial landmark extraction : Methods • Facial landmark extraction : Results • Future Steps

3. Research Problems “Automated recognition of facial expressions and identity” • Face identification • Recognition of facial expressions

4. Research Problems Face identification • Classification of input face to one of existing face classes stored in database • Rejection of input face as unrecognized/unknown face Face database Face classes Input face still image/video signal 1 … N Person A Recognition system classification … … … … rejection Person Z Unrecognized face

5. Research Problems Recognition of facial expressions • Facial expressions affect face recognition because a variability of facial landmarks in their appearance is high • Humans are good in recognizing facial identity regardless of changes in facial expressions • Computer-aided systems of face recognition are dramatically compromised by changes in facial expressions

6. The primary aim of this research is to do theoretical and experimental investigation on the possibilities to automatically recognize facial identity independent of changes in facial expressions For that purpose two 2D recognition systems will be developed Recognition system of facial expressions Expression-invariant system of facial identity recognition Aim of research

7. Extraction and classification of facial landmarks, namely, regions of eyes/eye-brows, nose, and mouth from still images Detection and recognition of facial expressions - how facial muscle activations can change appearance of a face during emotional reactions? Expression-invariant recognition of facial identity Tasks

8. Methods of landmarkextraction Feature-b ased method • Uses knowledge on geometrical structure of human faces • Based on geometrical features of facial landmarks, such as position of eyes/eye-brows, nose, and mouth • Four regions of interest have been selected as most informative for further recognition steps • right eye-brow / eye • left eye-brow / eye • nose • mouth

9. Methods of landmarkextraction 0 15 Template-b ase method 1 14 2 13 3 22.5° 12 4 • Represents a face as a feature map/template of original facial image • Local oriented edges are used to construct a feature map of the facial image • Orientation of edges has been determined with step of 22.5 and encoded as 0,1,…..15 11 5 10 6 9 7 8 Oriented edges extracted in left eyeregion

10. Methods of landmarkextraction Database • Tests were performed using Pictures of Facial Affect [1] • 110 images with 7 basic facial displays: happiness, surprise, fear, anger, disgust and neutral expression • Images were first normalized to three pre-set sizes 100X150, 200X300 and 300X400 in order to test the effect of image size to the operation of the algorithms • In sum 110 x 3 = 330 images were used for algorithm testing [1] Ekman, P., Friesen, W. V., & Hager, J.C. (2002) Facial Action Coding System (FACS). Published by A Human Face, Salt Lake City, UTAH: USA

11. Facial landmarkextraction Pre-processing algorithms • RGB – grey-level transformation • Multiresolution image representation was performed using a recursive Gauss transformation Different resolution levels Resolution level 0 Resolution level 2 Transformation Input face (RGB) Normalization (grey-level scale)

12. Facial landmarkextraction • Final feature map has been constructed on base of local oriented edges extracted in each point of grey-level image at each resolution level with exception of points which had the contrast values less than threshold • Extraction of local edges has been performed by calculation of difference between two oriented Gaussians with shifted kernels, which allows determining both orientation and contrast of local edge Points of interest have been grouped - if the distance between points of interest was less than the threshold the points were grouped - otherwise ignored Final feature map Map of detected points of interest

13. Orientation portraits of the facial landmarks Right Eye Left Eye Number of points of interest Nose Mouth Edge orientation Facial landmarkextraction • Detected regions of interest have been compared with orientation portraits of facial landmarks I have constructed earlier • Regions which did not correspond to the portraits have been ignored • Pre-knowledge about facial structure have been used matching

14. Facial landmarkextraction Finally, facial landmarks have been detected! neutral disgust fear Examples of feature maps of high-contrast oriented edges detected from the expressive images

15. Results Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes

16. Results Performance of the facial landmark detection algorithm averaged by all expressions for three image sizes

17. Results (a) (b) Right Eye Left Eye (c) (d) Nose Mouth Performance of the feature detection system for three image sizes. N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust

18. Results (a) (b) Right Eye Left Eye (c) (d) Nose Mouth Performance of the feature detection system for three image sizes. N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust

19. Results Problems • Algorithms are slow – about few seconds • Errors in groupping points of interest (red rectangles a, b, c) • Some landmarks are undetectable (d) (a) (b) (c) (d)

20. Results Recommendations • To improve detection of nose and mouth regions two alternatives are proposed • The first one is selection of different thresholds for detection and groupping of points of interest for different resolution levels • The second alternative requires more careful processing of detected regions and searching different landmark parts such as eye and mouth corners and nostrils.

21. Future steps • Full article about automated expression-invariant detection of facial landmarks; short article about how emotions affect cognitive functioning and how this knowledge might be implicated for HCI • To improve landmark detection; implement prototype of 2D recognition system of facial expressions (iExpRec) • To implement and test iExpRec • To implement of expression-invariant 2D facial identity recognition system (iFaceRec).

22. Thank you f r your attention!