Perceptually Guided Expressive Facial Animation

1. Perceptually Guided Expressive Facial Animation Zhigang Deng and Xiaohan Ma Computer Graphics and Interactive Media Lab Department of Computer Science University of Houston http://graphics.cs.uh.edu

2. Talk Outline • Motivation • Related Work • Our Approach • Construction of facial perceptual metric • Perceptually Guided Facial Animation Algorithms • Results and User Studies • Conclusions and Discussion

3. Motivations • How to efficiently measure and synthesize realistic expressive facial animation? • The ultimate measuring stick is human perception. • Intrinsic sensitivity to the subtlety of animated faces • Current popular means is to conduct subjective user studies (offline, post-production evaluation tool) • Not automated (tedious human involvements) • Inefficient (time-consuming experiment setup and user study) • Costly (participant cost)

4. Related Work • Facial animation techniques • Geometric deformation [Singh and Fiume 98, Noh and Neumann 01,Sumner and Popovic 04], physically-based [Lee et al 95, Sifakis et al. 05], performance driven facial animation [Williams 90], facial expression synthesis and editing [zhang et al. 03, Joshi et al. 03] • Data-driven approaches for facial animation [Brand 99, Chuang et al. 02, Vlasic et al. 05,Wampler et al. 07,Bregler et al. 97, Kshirsagar and Thalmann 03, Cao et al. 04, Deng and Neumann 06] • Focus on the mathematical accuracy/efficiency of their algorithms • Little attention has been paid to perceptual aspects to their algorithms • Automatic analysis of facial expressions in computer vision community [Pantic and Rothkrantz 00, Tian et al. 01, Valstar and Pantic 06] • Focus on analysis side of facial expressions

5. Related Work • User studies for character animation [Hodgins et al. 98, Sullivan and Dingliana 01, Sullivan et al. 03, Watson et al. 01, Reitsma and Pollard 03, Wang and Bodenheimer 04, McDonnell et al. 06, McDonnell et al. 07] • Measure the association between human perception and factors of character animation • Subjective evaluation have been also conducted to gain human perceptual insight on facial animation [Cunningham et al. 03, Cunningham et al. 04, Wallraven et al. 05, Wallraven et al. 08, Geiger et al. 03, Cosker et al. 05] • Most of these efforts are centered at the qualitative side • Our work • Aim to quantitatively model (perceptual metric) the association between facial motion and perceptual outcomes • Further exploit this perceptual metric to build perceptually guided expressive facial animation algorithms

6. Our Work • A novel computational facial perceptual metric (FacePEM) • Measuring and predict the expressiveness (type/scale) of synthetic expressive facial animations • Learn statistical perceptual prediction model (FacePEM) which can measure and predict the perceptual outcomes of arbitrary facial motion • FacePEM-guided facial animation algorithms • Perceptual metric-guided speech animation synthesis • Expressive facial motion editing enhanced with expressiveness cues

7. Construction of FacePEM

8. Data Acquisition • Optical motion capture system for facial motion capture • 103 facial markers (95 face markers, 4 head markers, and 4 neck markers) • Four human subjects spoke sentences with four emotions (happy, angry, sad, and neutral) • Remove head motion and align facial motions of different subjects

9. Construction of FacePEM

10. User Evaluation Study • Participants were required to identify perceived emotion and corresponding emotional expressiveness scale (1 to 10)as a nonforced-choice taskafter viewing each clip. • 68 facial motion clips, 30 participants (university students) • Perceptual Outcome Vector (POV) for each facial motion clip

11. Facial Motion Analysis • Face segmentation • PCA is a global transformation; no explicit correspondence between PCA eigen-vectors and localized facial movements. • Physically-motivated segmentation [Joshi et al. 03]: six regions (forehead, eye, left cheek, right cheek, mouth, nose) • Region-based motion reduction • Apply PCA for the movements of each region • Region-based PCA eigen-vectors correspond to meaningful, localized facial movements [Li and Deng 07]. The first/second largest eigen-vectors of themouth region The first/second largest eigen-vectors of the eye region

12. Facial Motion Modeling • Modeling region-based expressive facial motion patterns • Use the M-order Linear Dynamical Systems (LDS) [Pavlovic et al. 00, Chai and Hodgins 07] to model region-based facial motion patterns for any specific emotion • 18 LDSs are fitted (6 regions*3 emotions), xn= ∑(Ai*xn-i)+vi • Objective Matchness Vector (OMV) • Closeness function: Describes the closeness of any given facial motion sequence at a specific facial region representing a specific emotion • A OMV vector encloses 18 components Pemo,reg (Si) = e-E E = -ln F(S) = -ln F(x1:T) ≈ C*∑||xt-∑(Ajxt-j)-vj||2

13. Construction of FacePEM

14. Learning Perceptual Prediction • Statistical perceptual prediction model • Predict its POV, based on the OMV of input facial motion sequence • Three approaches: Least-square based linear fitting, Radial Basis Functions (RBFs) network approach, and Support Vector Machines (SVMs) • SVMs-based perceptual prediction model achieve the minimum error. • Cross-Validation for test and validation • 54 facial motion clips as the training dataset • 14 facial motion clips as the test/validation dataset

15. Review of FacePEM Framework

16. Perceptually Guided Facial Animation Algorithms • Perceptually guided expressive speech animation synthesis • The core part of many data-driven speech animation synthesis [Brelger et al. 97, Kshirsagar and Thalmann 03, Cao et al. 04, Deng and Neumann 06] • Cost = PhoMtchCost+ConstrCost+SmoCost • More intelligent and smart “cost function” with perceptual metric • Expressive facial motion editing enhanced with expressiveness cues • Current facial motion editing techniques [Chuang et al. 02, Cao et al. 03, Joshi et al. 03, Vlasic et al. 05, Li and Deng 07] do not provide feedback or expressiveness cues • FacePEM will measure and display its updated emotion type and expressiveness scale to users in a timely means.

17. Perceptually Guided ExpressiveSpeech Animation Synthesis • Choose an expressive speech animation synthesis algorithm [Deng and Neumann 06] for test/validation • Predict emotion type and expressiveness scale of facial motion sequence being synthesized, and Incorporate emotion scale into synthesis algorithm • EC(s,Emo)=C*(1-Same(Emo, EmoLabel(s))) • EC(s, Emo) = C*(1-CalcEmo(s)[Emo])

18. Facial Motion Editing Enhanced with Expressiveness Cues • Choose an expressive facial motion editing system for validation [Li and Deng 07] • Predict emotion type and expressiveness scale of the edited facial motion sequence • Display expression type/scale to users as feedback

19. Results & Evaluation - I • 20 expressive facial animation clips (10 from this work, 10 from previous work) • 20 participants rated the visual fidelity (expression) of these clips

20. Results & Evaluation - II • 10 expressive edited facial motion clips (5 from this work, 5 from previous work) • 20 participants rated the visual fidelity (expression) of these clips

21. Discussion • It is hard to know how much data would be enough to train well-behaviored statistical learning approaches • We did not consider the effects of eye gaze motion • Limited emotion types (3 basic emotion) are studied. • Idiosyncratic motion signals might exist in current used facial motion dataset.

22. Conclusions • We present a novel computational perceptual metrics for measuring and predict the expression (type/scale) of facial animations • Bridge human perceptual insights with objective facial motion patterns • Demonstrated perceptually guided expressive facial animation algorithms on two cases • Expressive speech animation synthesis • Interactive expressive facial motion editing • Future Work • Study more data and expression types • Remove the idiosyncratic components from the data

23. Questions?