Human Emotion Synthesis

1. Human Emotion Synthesis David Oziem, Lisa Gralewski, Neill Campbell, Colin Dalton, David Gibson, Barry Thomas University of Bristol, Motion Ripper, 3CR Research

2. Project Group • Motion Ripper Project • Methods of motion capture. • Re-using captured motion signatures. • Synthesising new or extend motion sequences. • Tools to aid animation. • Collaboration between University of Bristol CS, Matrix Media & Granada. Synthesising Facial Emotions – University of Bristol – 3CR Research

3. Introduction • What is an emotion? • Ekman outlined 6 different basic emotions. • joy, disgust, surprise, fear, anger and sadness. • Emotional states relate to ones expression and movement. • Synthesising video footage of an actress expressing different emotions. Synthesising Facial Emotions – University of Bristol – 3CR Research

4. Synthesising Facial Emotions – University of Bristol – 3CR Research

5. Video Textures • Video textures or temporal textures are textures with motion. (Szummer’96) • Schodl’00, reordered frames from the original to produce loops or continuous sequences. • Doesn’t produce new footage. • Campbell’01, Fitzgibbon’01, Reissell’01, used Autoregressive process (ARP) to synthesis frames. Examples of Video Textures Synthesising Facial Emotions – University of Bristol – 3CR Research

6. Autoregressive Process • Statistical model • Calculating the model involves working out the parameter vector (a1…an) and w. • n is known as the order of the sequence. Current value at time t y(t) = – a1y(t – 1) – a2y(t – 2) – … – any(t – n) + w.ε Parameter vector (a1,…,an) Noise Synthesising Facial Emotions – University of Bristol – 3CR Research

7. Autoregressive Process • Statistical model • Increasing dimensionality of y drastically increases the complexity in calculating (a1…an). y(t) = – a1y(t – 1) – a2y(t – 2) – … – any(t – n) + w.ε Synthesising Facial Emotions – University of Bristol – 3CR Research

8. Autoregressive Process Secondary mode Primary mode PCA analysis of Sad footage in 2D • Principal Components Analysis is used to reduce number of dimensions in the original sequence. Synthesising Facial Emotions – University of Bristol – 3CR Research

9. Autoregressive Process Secondary mode Secondary mode Primary mode Primary mode PCA analysis of Sad footage in 2D Generated sequence using an ARP • Non-Gaussian Distributionis incorrectly modelled by an ARP. Synthesising Facial Emotions – University of Bristol – 3CR Research

10. Face Modelling • Campbell’01, synthesised a talking head. • Cootes and Talyor’00, combined appearance model. • Isolates shape and texture. • Requires labelled frames. • Must label important features on the face. Labelled points Synthesising Facial Emotions – University of Bristol – 3CR Research

11. Combined Appearance Shape space Hand Labelled video footage provides a point set which represents the shape space of the clip. Synthesising Facial Emotions – University of Bristol – 3CR Research

12. Combined Appearance Shape space Texture space Warping each frame into a standard pose, creates the texture space. The standard pose is the mean position of the points. Synthesising Facial Emotions – University of Bristol – 3CR Research

13. Combined Appearance Shape space Texture space Combined space Joining the shape and texture space and then re-analysing using PCA produces the combined space. Synthesising Facial Emotions – University of Bristol – 3CR Research

14. Combined Appearance Shape space Texture space Combined space Reconstruction of the original sequence from the combined space. Combined space Synthesising Facial Emotions – University of Bristol – 3CR Research

15. Combined Appearance Secondary mode Secondary mode Primary mode Original sequence in 2D Change in distribution after applying The combined appearance technique Primary mode Combined Appearance sequence Synthesising Facial Emotions – University of Bristol – 3CR Research

16. Combined Appearance • Visually the generated plot appears to have been generated using the same stochastic process as the original. Secondary mode Secondary mode ARP model Primary mode Primary mode Generated Sequence Original sequence Synthesising Facial Emotions – University of Bristol – 3CR Research

17. Copying and ARP • Combine the benefits of copying with ARP • New motion signatures. • Handles non-Gaussian distributions. Synthesising Facial Emotions – University of Bristol – 3CR Research

18. Important to reduce the complexity of the search process. Need around 30 to 40 dimensions in this example. Copying and ARP Original input PCA Reduced input Synthesising Facial Emotions – University of Bristol – 3CR Research

19. Temporal segments of between 15 to 30 frames. Need to reduce each segment to be able to train ARP’s. Copying and ARP Original input PCA Segmented input Reduced segments PCA Reduced input Synthesising Facial Emotions – University of Bristol – 3CR Research

20. Many of the learned models are unstable. 10-20% are usable. Copying and ARP Original input PCA Segmented input Reduced segments PCA ARP Reduced input Synthesised segments Synthesising Facial Emotions – University of Bristol – 3CR Research

21. Copying and ARP Original input PCA Segmented input Reduced segments PCA ARP Reduced input Segment selection Synthesised segments Outputted Sequence Synthesising Facial Emotions – University of Bristol – 3CR Research

22. Example First mode Possible segments. End of generated sequence. Compared section Time t Synthesising Facial Emotions – University of Bristol – 3CR Research

23. Example First mode Closest 3 segments are chosen. Time t Synthesising Facial Emotions – University of Bristol – 3CR Research

24. Example First mode The segment to be copied is randomly selected from the closest 3. Time t Synthesising Facial Emotions – University of Bristol – 3CR Research

25. Example First mode Segments are blended together using a small overlap and averaging the overlapping pixels. Time t Synthesising Facial Emotions – University of Bristol – 3CR Research

26. Copying and ARP • Potentially infinitely long. • Includes new novel motions. Secondary mode Secondary mode Copying & ARP model Primary mode Primary mode PCA analysis of Sad footage in 2D Generated sequence Synthesising Facial Emotions – University of Bristol – 3CR Research

27. Results (Angry) • Combined appearance produces higher resolution frames. • Better motion from the copying and ARP approach Source Footage Combined Appearance ARP Copying with ARP Synthesising Facial Emotions – University of Bristol – 3CR Research

28. Results (Sad) • Similar results as with the angry footage • Copied approach is less blurred due to the reduced variance. Source Footage Combined Appearance ARP Copying with ARP Synthesising Facial Emotions – University of Bristol – 3CR Research

29. Comparison Results • Simple objective comparison. • Randomly selected temporal segments. - Combined appearance - Segment copying Synthesising Facial Emotions – University of Bristol – 3CR Research

30. Comparison • Perceptually is it better to have good motion or higher resolution. Synthesising Facial Emotions – University of Bristol – 3CR Research

31. Combined appearance Segment Copying with ARP Synthesising Facial Emotions – University of Bristol – 3CR Research

32. Other potential uses • Self Organising Map • Uses combined appearance • as each ARP model provides a minimal representation of the given emotion. • Can navigate between emotions to create new interstates. Angry SadHappy Synthesising Facial Emotions – University of Bristol – 3CR Research

33. Conclusions • Both methods can produce synthesised clips of a given emotion. • Combined appearance produces higher definition frames. • Copying and ARPs generates more natural movements. Synthesising Facial Emotions – University of Bristol – 3CR Research

34. Questions Synthesising Facial Emotions – University of Bristol – 3CR Research