Emotional Databases at TAU

1. Emotional Databases at TAU Noam Amir Samuel Ron Vered Aharonson

2. Method #1:Event Recollection

3. Underlying principles: • Event recollection method is based on • The ability to recollect an event • The ability to experience an emotional event • The ability to imagine the event • Subjects try to place themselves in emotional situations they had previously experienced • Assumption: under the correct conditions, most people can re-experience emotional episodes ( not just recollect them)

4. Conditions: • Quiet, comfortable, distraction free environments • Measurements: • Speech • Physiology: GSR, heart rate, muscle activity

5. Procedure: • Prerequisite to data collection: • There are subjects that either can’t recall an emotional event, or do not want to. They should not be examined. • 3. Subject should be aware of what is required from them. • 4. Subjects’ emotional state is crucial for reliable data.

6. The proper connections of the electrodes should be tested before the experiment starts.6. Experimental conditions should be comfortable. 7. The duration of the trial should not be told to the subject. 8. Subjects should be asked if they have any questions before the start of the experiment.9. Instructions should be read, preferable given by a tape recording.10. Eyes open/shut condition shouldn’t change during the experiment (recall is easier with eyes shut)11. Between emotions, subjects should be distracted in order to disengage them from previous emotion

7. 12. Test duration depends on physiological variables to be recorded: - some variables have longer time constants

8. During the experiment:1. The order of recalled emotions should be randomized. Always start with a “baseline” that should be repeated at the end of the experiment. 2. Recording always starts about 15-20 sec. after the subject started to describe. 3. If data is “noisy”, restart the recording but do not tell the subject to start the description of the recalled event again.

9. After the experiment:1. Evaluate whether subjects’ sounded emotional as expected (“participant”), or expressed the emotion as an outsider (an “observer”, more like narrator).2. Compare the evaluation of the subject on his expressed emotion with human judgments on: 1) scoring of emotional classification and degree. 2) Did the subject “switch” between emotional and non-emotional states? 4) Did the subject express the claimed emotion?

10. Advantages/drawbacks • This method does not include human-human interaction • Subjects can tend to switch between participant/observer states • The database is very large, and emotional content can be sparse • Emotion can be pervasive but subdued – more mood than bursts • Recalled emotion is considered a very effective method for producing emotional states

11. Database CullingWheat vs. Chaff

12. The raw event recollection database: • 150 speakers *7 emotions of 1 minute each = 17.5 hours of speech! • listening/judging is very lengthy! • Problems: • Emotional content fluctuates • Text is indicative towards emotion • Speakers switch between “participant” and “observer” states

13. An initial listening test: • For 30 speakers, 6 judges were asked to listen to the unedited recordings • For each separate minute, judges were required to fill the complex questionnaire shown on next slide • Note:the fully automated test was implemented in MATLAB. Very convenient!

15. Conclusions from initial test: • A full test took about 10 (non-consecutive!) hours • Time consuming • Expensive • Still leaves us with uncertainty as to where the “real” emotional parts reside • Valuable information is mainly in – • Percentages • Credibility ratings

16. Cullinga smaller database from the large one: • Our objective is to extract from the recordings a smaller database – • Easier to subject to listening tests • Easier to analyze acoustically • Contains the parts that are truly emotional • Hopefully has more neutral textual content

17. How to do it? • Simplest answer: Use students! • Other possibilities: • Use automated methods (Nick Campbell does this) • Use researchers ( Puh-lease…)

18. “Student” method: • We let two students listen to the recordings rated most highly for credibility • They were asked to extract about 20 ten-second excerpts for each emotion • They were also requested to try for utterances where the textual content was as neutral as possible • The excerpts were subjected to a second listening test

19. Second listening test: • Far simpler • About one hour per listener (in two half-hour sessions) • Objectives: • To classify and rate emotional content • To verify “participant” status of speaker • To examine the influence of textual content • Subjects: 19 young female students

20. Questionnaire:

21. Questionnaire! • What is the expressed emotion? (anger, joy, sadness, fear disgust, neutral, other) • What is the degree of emotion? (weak, medium, strong) • In your opinion, is the speaker really experiencing the said emotion? (yes, no) • Did the textual content influence your decision? (yes, no) • Did you deliberate between the chosen emotion and another? (yes, no)If yes – which? (anger, joy, sadness, fear disgust, neutral, other)

22. Results and Conclusions • The results are being analyzed statistically • Some anecdotal results and examples follow

23. Can listeners really judge whether the textual content is relevant? • Recording: • The emotion is: Sadness (sad67.wav) • Text:“the whole family … and his children” • Yet 9 listeners said the text influenced their judgment! • Recording: • The emotion is: Sadness (sad25.wav) • Text:“And I returned from a trip outside with my niece, and I came in and my mother was sitting inside…” • 14 listeners judged the emotion as sadness13 of the listeners said the text influenced their judgment!

24. Content can be misleading, too: • Recording: • The emotion is: Joy (Joy52.wav) • Text:“you want to shout and you shout till it comes out as…till you all get to …” • 9 listeners judged the emotion as fear13 of the listeners said the text influenced their judgment!

25. Overall: • Sadness was most easily identified • Disgust was often confused with Sadness or Fear • Fear was often confused with Sadness • Anger Joy and Neutral were mostly identifiable • Usually the degree of identification was correlated with the degree of emotion • Listeners are poor at judging the effect of text on their judgment • Strong prosodic indicators can convince them that the text is influential! • A separate text-only test should be conducted

26. Remark: • During data collection, subjects were asked to express five “basic” emotions • Labeling need not follow this scheme!

27. Method #2:Computer Gaming

28. Alternative approach: Games and simulators • These provide an extremely interesting setting • Participants became immersed in the game – our initial impression is that they experienced real emotions • The experimenter can control these to a certain extent

29. Our implementation: • Gambling game after Damassio’s experiment [Damasio, 1994] . • "real life" emotions • anticipation/apprehension • disappointment • contentment • voice-activated game: two phrases repeated ~100 times under different conditions. • Labeling according to Damassio’s event definitions

30. Our data acquisition method: based on Damassio’s experiment Gain:

31. Loss:

32. A movie sample….

33. Measured parameters: • Voice • Facial expressions • Mouse movements • GSR • ECG • PPG

34. Remarks: • Many repetitions of the same 2 phrases is an interesting method to examine prosodic variation • Listening tests can be very tiresome • We have an objective indication of the expected emotion • This method lends itself very well to multimodality • Real man-machine interaction

35. Results • Analysis is not complete yet • Initial classification using rule-based analysis shows encouraging results

36. Summary: • At TAU we’re currently using two main methods: • Event recollection • Computer gaming • We intend to use SAL type induction in the near future, in Hebrew • We have some limited experienced with media based data, which may be extended