Facial Expression Detection using PCA, a Thermal Approach

1. Facial Expression Detection using PCA, a Thermal Approach A brief foray into thermographic and visual imaging approaches. Robert S. Pienta

2. Table of Contents • Background • Methodology • Data Collection • Data Analysis • Results • Conclusion & Future Work

3. Background • Can thermography be used to recognize particular action units? • The heat maps from a face are unique • Overcomes many challenges of the visual approach as thermal does not rely on a light source [7,8,9] • The Facial Action Coding System (FACS) is a means by which to break down facial expressions into Action Units, each based off of muscles [1]. • A qualitative taxonomy of facial expressions

4. Methodology To capture individual action units a set of regions of interest (ROIs) were selected. These ROIs are placed at very particular locations on the face. Each ROI tracks the contraction of a muscle by detecting the tissue deformation via changes in the heat map.

5. Data Collection Experimental setup. • Three intensities of lighting were used during the tests, low, medium, and high intensity fluorescent light. This was to broaden the variance in lighting as recommended in [5]. • Data were collected on 7 initial test subjects in thermal and visual simultaneously • Subjects had a wide variety of appearance • Gender • Ethnicity • Facial hair

6. Data Analysis For each ROI: … n k 1 m nm nm … Frame: 1 … k, where k is the final frame. • Principal component analysis (PCA) is used to extract the vectors that correspond with the greatest change. • This allows us to detect the frames of greatest change for each ROI. Each frame of the video is added to a matrix by vectorizing it.

7. PCA Output for ROI IX, Zygomaticus Muscle PC Coefficients Frame The coefficients of the first principle components, denoting change in a ROI over time, as expressions are formed. This denotes change in just one ROI over the total number of frames.

8. PCA Signal Analysis We then calculate the standard deviation of the PCA output for the course of an expression. As we have 13 different regions, we create an expression profile for that expression.

9. Two Sample Profiles

10. Classification • A number of different machine learning approaches were utilized: • Initial tests were done using Weka 3: data mining software. • Further experimentation with neural nets was done in MATLAB.

11. Results • The best results were from a multilayer perceptron

12. Process Overview Place 13 Fundamental ROIs Slice every frame for each ROI into a vector Expression Data ROI Frame Sequences Standard deviation of PCA output for each expression Combine the vector representation of each frame into a matrix Apply PCA to each ROI matrix PC ROI matrices Machine learning based classification Expression Profile

13. Conclusions and Future Work • Thermal data can be used to isolate the activation of particular facial muscles • Areas with low tissue deformation won’t be detected in the visual, but may still in the thermal due to the non-uniform heat distribution • Further investigate ways to capture orbicularis oculi (exterior eye) contractions • Use tracking to make a robust system to analyze movement and expressions • Fully automate placement of ROIs • Expand the number of subjects drastically

14. Questions? References [1] Ekman, P. and Friesen, W. ; Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, 1978. [2] Kotsia, I.; Pitas, I.; , "Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines," Image Processing, IEEE Transactions on , vol.16, no.1, pp.172-187, Jan. 2007 [3] Zeng, Z.; Fu, Y.; Roisman, G. ; Wen, Z.; Hu, Y;. Huang, T. ;, “Spontaneous Emotional Facial Expression Detection” Journal of Multimedia, vol. 1, pp.1-8, 2006 [4] Whitehill, J.; Littlewort, G.; Fasel, I.; Bartlett, M.; Movellan, J.; , "Toward Practical Smile Detection," Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.31, no.11, pp.2106-2111, Nov. 2009 [5] T. Sim, S. Baker, and M. Bsat, “The CMU Pose, Illumination, and Expression Database,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 12, pp. 1615-1618, Dec. 2003. [6] Fasel, B.; Luettin, J.; Automatic facial expression analysis: a survey, Pattern Recognition, Volume 36, Issue 1, January 2003, Pages 259-275, ISSN 0031-3203, DOI: 10.1016/S0031-3203(02)00052-3. [7] Black, M. ; Yacoob, Y. ; “Recognizing facial expressions in image sequences using local parameterized models of image motion”,International Journal of Computer Vision 25 (1) (1997) 23–48. [8] Jarlier, S.; Grandjean, D.; Delplanque, S.; N’Diaye, K.; Cayeux, I.; Velazco, M.; Sander, D.; Vuilleumier, P.; Scherer K.; “Automatic Thermal Analysis of Facial Expressions” IEEE Transactions on Affective Computing. Mar. 2010 [9] Yoshitomi, Y.; Miyawaki, N.; Tomita, S.; Kimura, S.; , "Facial expression recognition using thermal image processing and neural network," Robot and Human Communication, 1997. RO-MAN '97. Proceedings., 6th IEEE International Workshop on , vol., no., pp.380-385, 29 Sep-1 Oct 1997