Facial Expressions in Portuguese Sign Language Videos

1. Facial Expressions in Portuguese Sign Language Videos André Maia

2. Index • Motivation • Why Facial Expressions? • Problem Characterization • The Process: • Facial Expressions Analysis • Face Databases • Face Detection • Feature Extraction • Classification • Work plan Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 1

3. Motivation • How can weallcommunicatethesameway? • How can wefind a solution for thosewhodon’tknowsignlanguage? Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 2

4. Why Facial Expressions? • Facial and/or body expressivity is fundamental in any sign language. • It is necessary to demonstrate the feeling to the given gesture and sometimes affecting its meaning Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 3

5. Why Facial Expressions? • Can mean interrogation, exclamation, denial, statement, doubt, shame, fear, sadness, etc.… • There are gestures that can only be distinguished by the facial expression or the vocal sound that accompanies Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 4

6. Problem Characterization • Given a recognition problem, how can we identify the expression? • How can we compare it? Are we correct? If not, how close are we? • How can one agent (algorithm) try to understand its result? Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 5

7. Problem Characterization • Facial Expressions Analysis • Face Databases • Face Detection • Feature Extraction • Classification Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 6

8. The Process – Facial Expressions Analysis • Facial Action Coding System • Prototypic Emotional Expressions Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 7

9. The Process – Facial Expressions Analysis • Facial Action Coding System • Describes people’s expressions • in terms of action units • Does not include any inferential • labels • Prototypic Emotional Expressions Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 8

10. The Process – Facial Expressions Analysis • Facial Action Coding System • Prototypic Emotional Expressions • Majority of facial expression recognition system attempt to recognize a small set of emotions • 1 - Disgust, 2 - Fear, 3 - Joy • 4 - Surprise, 5 - Sadness, • 6 – Anger Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 9

11. The Process – Face Databases • Large number of face databases have been collected Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 10

12. The Process – Face Databases • It is essential to have test data to check new approaches and to compare them with previous systems Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 11

13. The Process – Face Databases • Labellingthe media. • Different human coders means different labelling • Reduce subjectivity, using coding systems (FACS) Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 12

14. The Process – Face Detection • Face detection is still an ongoing area • It is the first step in face recognition • Skin Color, motion, • facial/head shape, facial appearance • Most successful are based • on appearance Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 13

15. The Process – Face Detection • Eigenface and Template Matching • Viola’s Framework Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 14

16. The Process – Feature Extraction • Still Image based methods • For both images and videos • Video frames considered independently • Video based methods • Only for video • Motion information considered Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 15

17. The Process – Feature Extraction • Still Image based methods • Facial feature as graph deformation • Furrow presence detection • Comparison with reference face image • Video based methods Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 16

18. The Process – Feature Extraction • Still Image based methods • Recognize facial features: • Color information • Edge information • Shape information • Recognize furrows: • Edge information • Texture information • Video based methods Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 17

19. The Process – Feature Extraction • Still Image based methods • Video based methods • Motion analysis: Optical flow, tracking algorithms • (Kalman, Condensation,…) • Active Apearence Models (AAM) Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 18

20. The Process – Feature Extraction Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 19

21. The Process – Classification • Wide range of classifiers used in automatic expression recognition • Bayesian, thresholding the Normalized Euclidian distance, Fisher discrimination analysis, Local Linear Embedding, Higher Order Singular Value Decomposition • Neural Network and Support Vector Machines Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 20

22. Work Plan • Analyze and decide which Database to use • Cohn-Kanadedatabase, Japanese Female Facial Expression (JAFFE) database, MMI database, and CMU-PIE database • Selection of the facial expression recognition algorithm • Implementation of the face detection algorithms • Proposal of a new method Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 21

23. Questions? Facial Expressions Analysis Why Facial Expressions? Problem Characterization Face Databases Face Detection Feature Extraction Work Plan Classification Index Motivation 22