facial expression detection using pca a thermal approach n.
Skip this Video
Loading SlideShow in 5 Seconds..
Facial Expression Detection using PCA, a Thermal Approach PowerPoint Presentation
Download Presentation
Facial Expression Detection using PCA, a Thermal Approach

Loading in 2 Seconds...

play fullscreen
1 / 14

Facial Expression Detection using PCA, a Thermal Approach - PowerPoint PPT Presentation

  • Uploaded on

Facial Expression Detection using PCA, a Thermal Approach. A brief foray into thermographic and visual imaging approaches. Robert S. Pienta. Table of Contents. Background Methodology Data Collection Data Analysis Results Conclusion & Future Work. Background.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Facial Expression Detection using PCA, a Thermal Approach' - felix-serrano

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
facial expression detection using pca a thermal approach

Facial Expression Detection using PCA, a Thermal Approach

A brief foray into thermographic and visual imaging approaches.

Robert S. Pienta

table of contents
Table of Contents
  • Background
  • Methodology
    • Data Collection
    • Data Analysis
  • Results
  • Conclusion & Future Work
  • Can thermography be used to recognize particular action units?
    • The heat maps from a face are unique
    • Overcomes many challenges of the visual approach as thermal does not rely on a light source [7,8,9]
  • The Facial Action Coding System (FACS) is a means by which to break down facial expressions into Action Units, each based off of muscles [1].
    • A qualitative taxonomy of facial expressions

To capture individual action units a set of regions of interest (ROIs) were selected.

These ROIs are placed at very particular locations on the face.

Each ROI tracks the contraction of a muscle by detecting the tissue deformation via changes in the heat map.

data collection
Data Collection

Experimental setup.

  • Three intensities of lighting were used during the tests, low, medium, and high intensity fluorescent light. This was to broaden the variance in lighting as recommended in [5].
  • Data were collected on 7 initial test subjects in thermal and visual simultaneously
  • Subjects had a wide variety of appearance
    • Gender
    • Ethnicity
    • Facial hair
data analysis
Data Analysis

For each ROI:







Frame: 1 … k, where k is the final frame.

  • Principal component analysis (PCA) is used to extract the vectors that correspond with the greatest change.
  • This allows us to detect the frames of greatest change for each ROI.

Each frame of the video is added to a matrix by vectorizing it.

pca output for roi ix zygomaticus muscle
PCA Output for ROI IX, Zygomaticus Muscle

PC Coefficients


The coefficients of the first principle components, denoting change in a ROI over time, as expressions are formed. This denotes change in just one ROI over the total number of frames.

pca signal analysis
PCA Signal Analysis

We then calculate the standard deviation of the PCA output for the course of an expression.

As we have 13 different regions, we create an expression profile for that expression.

  • A number of different machine learning approaches were utilized:
  • Initial tests were done using Weka 3: data mining software.
  • Further experimentation with neural nets was done in MATLAB.
  • The best results were from a multilayer perceptron
process overview
Process Overview

Place 13 Fundamental ROIs

Slice every frame for each ROI into a vector

Expression Data

ROI Frame Sequences

Standard deviation of PCA output for each expression

Combine the vector representation of each frame into a matrix

Apply PCA to each ROI matrix


ROI matrices

Machine learning based classification

Expression Profile

conclusions and future work
Conclusions and Future Work
  • Thermal data can be used to isolate the activation of particular facial muscles
  • Areas with low tissue deformation won’t be detected in the visual, but may still in the thermal due to the non-uniform heat distribution
  • Further investigate ways to capture orbicularis oculi (exterior eye) contractions
  • Use tracking to make a robust system to analyze movement and expressions
  • Fully automate placement of ROIs
  • Expand the number of subjects drastically
questions references
Questions? References

[1] Ekman, P. and Friesen, W. ; Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, 1978.

[2] Kotsia, I.; Pitas, I.; , "Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines," Image Processing, IEEE Transactions on , vol.16, no.1, pp.172-187, Jan. 2007

[3] Zeng, Z.; Fu, Y.; Roisman, G. ; Wen, Z.; Hu, Y;. Huang, T. ;, “Spontaneous Emotional Facial Expression Detection” Journal of Multimedia, vol. 1, pp.1-8, 2006

[4] Whitehill, J.; Littlewort, G.; Fasel, I.; Bartlett, M.; Movellan, J.; , "Toward Practical Smile Detection," Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.31, no.11, pp.2106-2111, Nov. 2009

[5] T. Sim, S. Baker, and M. Bsat, “The CMU Pose, Illumination, and Expression Database,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 12, pp. 1615-1618, Dec. 2003.

[6] Fasel, B.; Luettin, J.; Automatic facial expression analysis: a survey, Pattern Recognition, Volume 36, Issue 1, January 2003, Pages 259-275, ISSN 0031-3203, DOI: 10.1016/S0031-3203(02)00052-3.

[7] Black, M. ; Yacoob, Y. ; “Recognizing facial expressions in image sequences using local parameterized models of image motion”,International Journal of Computer Vision 25 (1) (1997) 23–48.

[8] Jarlier, S.; Grandjean, D.; Delplanque, S.; N’Diaye, K.; Cayeux, I.; Velazco, M.; Sander, D.; Vuilleumier, P.; Scherer K.; “Automatic Thermal Analysis of Facial Expressions” IEEE Transactions on Affective Computing. Mar. 2010

[9] Yoshitomi, Y.; Miyawaki, N.; Tomita, S.; Kimura, S.; , "Facial expression recognition using thermal image processing and neural network," Robot and Human Communication, 1997. RO-MAN '97. Proceedings., 6th IEEE International Workshop on , vol., no., pp.380-385, 29 Sep-1 Oct 1997