1 / 45

A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance

A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance. Steven R. Livingstone (BSc. BInfTech). Problem Statement. There exists no automated method to detect and influence the emotions of music

arion
Download Presentation

A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance Steven R. Livingstone (BSc. BInfTech) Slide 1 (of 45)

  2. Problem Statement • There exists no automated method to detect and influence the emotions of music • An audience’s response to computer music has previously been inaccessible to computers and thus lost Slide 2 (of 45)

  3. Hypothesis • Perceived emotional content of music can be influenced by controlling both the structural and performative aspects • Audience feedback can be captured by computers to tailor the musical experience Slide 3 (of 45)

  4. Methodology • Score – Manipulate the structure and mark up with emotional performance metadata • Audience – Determine emotional state and attitudes using affective computing tech. • Architecture – Bring together for an awareness of score and audience Slide 4 (of 45)

  5. Talk Overview • Introduction - In • Research and Contribution - Re • System & Testing - Sy • Future Work - Fu • Summary - Su Slide 5 (of 45)

  6. In 1 Why Emotions? • Principal target is the computer game • Emotional Narrative is the key to game enjoyment Slide 6 (of 45)

  7. In 2 Why Music? • Music is a universal human trait and found everywhere [a]: • cinema, television, radio, commercials, ballet, shopping centres, transport, waiting rooms, restaurants … • Within any waking 2 hour period, a person has a 44% chance of experiencing a musical event [b] Slide 7 (of 45)

  8. In 3 Why Gaming Music? • Gaming music is important for: emotion, interest and information • Game music static within scenes, unlike cinema music Slide 8 (of 45)

  9. In 4 How are we going to do it? • Cross-disciplinary approach to research • Study of emotion • Music psychology • Empirical Analysis • Bring this knowledge together into a computing framework Slide 9 (of 45)

  10. R 1 Emotion - History • 1600s – Primary Emotions (Descartes) • 1800s - Biological reaction mechanisms (Darwin) • 1800/1900s – Perception  Physiological Response  Emotion (James-Lange) • 1960s – Cognition and Appraisal Theory (Arnold and Lazarus) • 1990s – Somatic Markers (Damásio) Slide 10 (of 45)

  11. R 2 Emotion – Perceived VS Induced • Perceived Emotion – The emotion observer believes the source or stimuli is experiencing or expressing • Induced Emotion – The emotion felt by the observer as a result of the stimuli (very hard to capture/use) Fearful Speech Slide 11 (of 45)

  12. R 3 Emotion - Representation and Capture • Required for • Empirical analysis • Computational Implementation • Requirements • Continuous capture over time • Continuous representation of emotion (numerical) • Data Consistency Slide 12 (of 45)

  13. R 4 Emotion - Representation and Capture • Existing Methods Slide 13 (of 45)

  14. R 5 Emotion – Dimensional Approach • Originally proposed by Wilhelm Wundt in the 1800s • We choose 2D with Arousal & Valence • Arousal: Active  Passive • Valence: Positive  Negative • 2 dimensions offer a balance between ease of reporting and data richness [c] Slide 14 (of 45)

  15. R 6 Emotion - Representation and Capture • 2 Dimensional Emotion Space [d] Slide 15 (of 45)

  16. R 7 Music Emotion Rules • Need to influence the emotions somehow … • Over a century of empirical music psychology has investigated the link between music  emotion [e] • Two types • Structural – Modifying the score • Performative – Those applied by the performer when converting the score to audio Slide 16 (of 45)

  17. R 8 Music Emotion Rules – Structural • Structural Music Rules • An understanding of the musical structure • Simplified emotional grouping (octants) and testing • Varying degrees of musical theory required • A New Approach [1] Slide 17 (of 45)

  18. Slide 18 (of 45)

  19. R 10 Music Emotion Rules – Structural • Primary Music Emotion Structural Rules [2] • Can you hear them? Quad 1 (happy) Quad 2 (angry) Quad 3 (sad) Quad 4 (dreamy->bliss) Slide 19 (of 45)

  20. R 11 Music Emotion Rules – Performative • Performative Music Rules • Requires fine-grained, continuous capture of emotion for testing • Waveform modification (very complex stuff) • Already been done (partially) Slide 20 (of 45)

  21. R 12 Music Emotion Rules – Performative • Performance rules to accentuate emotion … • Chord asynchrony (melody lead or lag) • Rubato (especially at phrase boundaries) • Melody notes louder • Increase dynamic range (gradient) • Increase vibrato amplitude Structural Structural + Performative Slide 21 (of 45)

  22. R 13 Music Emotion Rules – Vocal • What does this table mean? Slide 22 (of 45)

  23. R 14 Music Tension and Induced Emotion • Is musical emotion really just 2 dimensional? • Perceived maybe, induced definitely not • Meyer believed that musical emotion is the inhibition or completion of musical expectations [f] “Tense Sad”, breaks the rules … important! Slide 23 (of 45)

  24. R 15 Audience Consideration • Awareness of Audience • Plays an important role in performances • Affective Computing • Attitudes • User state Slide 24 (of 45)

  25. R 16 Audience Consideration • Attitudes are a cognitive powerful tool: • Quickly categorise data • Influence decision making • Relatively static Slide 25 (of 45)

  26. R 17 Audience Consideration • User State • A listeners response to the stimulus • Guides the performer • Continuous feedback • Affective Computing • Research from MIT • Various mechanisms to detect AND affect Slide 26 (of 45)

  27. R 18 Research  Something Real • Phew! A lot of research • Many topics not examined today • … What were we trying to do again? Slide 27 (of 45)

  28. Hypothesis • Perceived emotional content of music can be influenced by controlling both the structural and performative aspects • Audience feedback can be captured by computers to tailor the musical experience Slide 28 (of 45)

  29. Sy 1 The Rule System – Influence • Influencing perceived emotions • E.g. make “happier” • How? • Apply octant-grouped structural rules • E.g. “Influence to be upbeat and positive” • Apply octant 2 rules (tempo [fast], loudness [loud] …) to music structure Slide 29 (of 45)

  30. Sy 2 The Rule System – Detection • Why? • Good influencing needs emotional context • Requires • Model of Music Tension • Advanced pattern matching • Advanced knowledge of music theory and composition • Very tricky … Not attempted before Slide 30 (of 45)

  31. Sy 3 The Architecture Slide 31 (of 45)

  32. Sy 4 The Architecture – Application Intent • Emotive Information • The [Arousal, Valence] vector [3] Slide 32 (of 45)

  33. Sy 5 The Architecture – Audience Sensing • Audience response provides a wealth of feedback data to performer • Attitudes and audience response [A, V] incorporated • E.g. Cap the fearfulness of a room’s music • Affective computing • Keystroke/mouse movement: Arousal and Tension • Gaze tracking/Skin conductivity: Arousal and Interest • Same problems as measuring induced emotion though Slide 33 (of 45)

  34. Sy 6 The Architecture – Emotive Algorithm • Equalising unit for [A, V] coming from game and audience • Player Cap: • Room Value: • Game Event: • Resulting [A, V]: Slide 34 (of 45)

  35. Sy 7 Testing Progress [2] • Aims • Influence the perceived emotions of music with primary music emotion structural rules • Rules can apply to both Western classical and standard computer game music • Testing • Listener played original work, followed by altered work (e.g. apply octant 2 rules) • How did emotion baseline change? • 11 participants, played 6 altered versions Slide 35 (of 45)

  36. Sy 8 Testing Progress • Overall Results • Looks OK but why the A, V discrepancy? Slide 36 (of 45)

  37. Sy 9 Testing Progress • Quadrant Breakdown Slide 37 (of 45)

  38. Sy 10 Testing Progress • Not Angrier, why? • Music selection (something deeper going on ..?) • Incomplete rule implementation for quadrant 2 Slide 38 (of 45)

  39. Methodology Recap • Score – Manipulate the structure and mark up with emotional performance metadata • How are we going? • Structure: Implemented and progressing • Performative: Identified, future implementation Slide 39 (of 45)

  40. Methodology Recap • Audience – Determine emotional state and attitudes using affective computing tech. • How are we going? • Identified and developed a theoretical implementation Slide 40 (of 45)

  41. Methodology Recap • Architecture – Bring together for an awareness of score and audience • How are we going? • Theoretical Implementation at present Slide 41 (of 45)

  42. Fu Future Work • Implement more structural music rules • Expanded testing regime • Implement performative rules • Begin testing of performative rules • Detection component • Incorporate Affective Computing elements Slide 42 (of 45)

  43. Su 1 Summary • There exists no automated method to detect and influence the emotions of music • We’re getting there • An audience’s response to computer music has previously been inaccessible to computers and thus lost • Theoretical, still Future work Slide 43 (of 45)

  44. Su 2 Questions? • Contact • srl@itee.uq.edu.au • http://itee.uq.edu.au/~srl • Papers • [1] "Playing with Affect: Music Performance with Awareness of Score and Audience", 2005, Australasian Computer Music Conference • [2] "Dynamic Response: Real-Time Adaptation for Music Emotion", 2005, Australasian Conference on Interactive Entertainment • [3] “Influencing the Perceived Emotions of Music with Intent”, 2005, Third International Conference on Generative Systems (in review) Slide 44 (of 45)

  45. Su 3 References • [a] Brown, S., B. Merker, and N.L. Wallin, An Introduction to Evolutionary Musicology, in The Origins of Music, S. Brown, B. Merker, and N.L. Wallin, Editors. 2000, MIT Press. • [b] Sloboda, J.A. and S.A. O'Neill, Emotions in Everyday Listening to Music, in Music and Emotion, theory and research. 2001, Oxford Press. p. 415-429. • [c] Russell, J.A., Measures of emotion., in Emotion: Theory research and experience., R.P.H. Kellerman, Editor. 1989, New York: Academic Press. p. 81-111. • [d] Schubert, E., Measurement and Time Series Analysis of Emotion in Music. 1999, University of New South Wales. • [e] Meyer, L.B., Emotion and Meaning in Music. 1956: The University of Chicago Press. Slide 45 (of 45)

More Related