a system for detecting and influencing the emotions of music in computer mediated performance n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance PowerPoint Presentation
Download Presentation
A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance

Loading in 2 Seconds...

play fullscreen
1 / 45

A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance - PowerPoint PPT Presentation


  • 73 Views
  • Uploaded on

A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance. Steven R. Livingstone (BSc. BInfTech). Problem Statement. There exists no automated method to detect and influence the emotions of music

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance' - arion


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a system for detecting and influencing the emotions of music in computer mediated performance

A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance

Steven R. Livingstone

(BSc. BInfTech)

Slide 1 (of 45)

problem statement
Problem Statement
  • There exists no automated method to detect and influence the emotions of music
  • An audience’s response to computer music has previously been inaccessible to computers and thus lost

Slide 2 (of 45)

hypothesis
Hypothesis
  • Perceived emotional content of music can be influenced by controlling both the structural and performative aspects
  • Audience feedback can be captured by computers to tailor the musical experience

Slide 3 (of 45)

methodology
Methodology
  • Score – Manipulate the structure and mark up with emotional performance metadata
  • Audience – Determine emotional state and attitudes using affective computing tech.
  • Architecture – Bring together for an awareness of score and audience

Slide 4 (of 45)

talk overview
Talk Overview
  • Introduction - In
  • Research and Contribution - Re
  • System & Testing - Sy
  • Future Work - Fu
  • Summary - Su

Slide 5 (of 45)

why emotions

In 1

Why Emotions?
  • Principal target is the computer game
  • Emotional Narrative is the key to game enjoyment

Slide 6 (of 45)

why music

In 2

Why Music?
  • Music is a universal human trait and found everywhere [a]:
    • cinema, television, radio, commercials, ballet, shopping centres, transport, waiting rooms, restaurants …
  • Within any waking 2 hour period, a person has a 44% chance of experiencing a musical event [b]

Slide 7 (of 45)

why gaming music

In 3

Why Gaming Music?
  • Gaming music is important for: emotion, interest and information
  • Game music static within scenes, unlike cinema music

Slide 8 (of 45)

how are we going to do it

In 4

How are we going to do it?
  • Cross-disciplinary approach to research
    • Study of emotion
    • Music psychology
    • Empirical Analysis
  • Bring this knowledge together into a computing framework

Slide 9 (of 45)

emotion history

R 1

Emotion - History
  • 1600s – Primary Emotions (Descartes)
  • 1800s - Biological reaction mechanisms (Darwin)
  • 1800/1900s – Perception  Physiological Response  Emotion (James-Lange)
  • 1960s – Cognition and Appraisal Theory (Arnold and Lazarus)
  • 1990s – Somatic Markers (Damásio)

Slide 10 (of 45)

emotion perceived vs induced

R 2

Emotion – Perceived VS Induced
  • Perceived Emotion – The emotion observer believes the source or stimuli is experiencing or expressing
  • Induced Emotion – The emotion felt by the observer as a result of the stimuli (very hard to capture/use)

Fearful Speech

Slide 11 (of 45)

emotion representation and capture

R 3

Emotion - Representation and Capture
  • Required for
    • Empirical analysis
    • Computational Implementation
  • Requirements
    • Continuous capture over time
    • Continuous representation of emotion (numerical)
    • Data Consistency

Slide 12 (of 45)

emotion representation and capture1

R 4

Emotion - Representation and Capture
  • Existing Methods

Slide 13 (of 45)

emotion dimensional approach

R 5

Emotion – Dimensional Approach
  • Originally proposed by Wilhelm Wundt in the 1800s
  • We choose 2D with Arousal & Valence
    • Arousal: Active  Passive
    • Valence: Positive  Negative
  • 2 dimensions offer a balance between ease of reporting and data richness [c]

Slide 14 (of 45)

emotion representation and capture2

R 6

Emotion - Representation and Capture
  • 2 Dimensional Emotion Space [d]

Slide 15 (of 45)

music emotion rules

R 7

Music Emotion Rules
  • Need to influence the emotions somehow …
  • Over a century of empirical music psychology has investigated the link between music  emotion [e]
  • Two types
    • Structural – Modifying the score
    • Performative – Those applied by the performer when converting the score to audio

Slide 16 (of 45)

music emotion rules structural

R 8

Music Emotion Rules – Structural
  • Structural Music Rules
  • An understanding of the musical structure
  • Simplified emotional grouping (octants) and testing
  • Varying degrees of musical theory required
  • A New Approach [1]

Slide 17 (of 45)

music emotion rules structural1

R 10

Music Emotion Rules – Structural
  • Primary Music Emotion Structural Rules [2]
  • Can you hear them?

Quad 1 (happy)

Quad 2 (angry)

Quad 3 (sad)

Quad 4 (dreamy->bliss)

Slide 19 (of 45)

music emotion rules performative

R 11

Music Emotion Rules – Performative
  • Performative Music Rules
  • Requires fine-grained, continuous capture of emotion for testing
  • Waveform modification (very complex stuff)
  • Already been done (partially)

Slide 20 (of 45)

music emotion rules performative1

R 12

Music Emotion Rules – Performative
  • Performance rules to accentuate emotion …
    • Chord asynchrony (melody lead or lag)
    • Rubato (especially at phrase boundaries)
    • Melody notes louder
    • Increase dynamic range (gradient)
    • Increase vibrato amplitude

Structural

Structural + Performative

Slide 21 (of 45)

music emotion rules vocal

R 13

Music Emotion Rules – Vocal
  • What does this table mean?

Slide 22 (of 45)

music tension and induced emotion

R 14

Music Tension and Induced Emotion
  • Is musical emotion really just 2 dimensional?
    • Perceived maybe, induced definitely not
  • Meyer believed that musical emotion is the inhibition or completion of musical expectations [f]

“Tense Sad”, breaks the rules … important!

Slide 23 (of 45)

audience consideration

R 15

Audience Consideration
  • Awareness of Audience
  • Plays an important role in performances
  • Affective Computing
  • Attitudes
  • User state

Slide 24 (of 45)

audience consideration1

R 16

Audience Consideration
  • Attitudes are a cognitive powerful tool:
    • Quickly categorise data
    • Influence decision making
    • Relatively static

Slide 25 (of 45)

audience consideration2

R 17

Audience Consideration
  • User State
    • A listeners response to the stimulus
    • Guides the performer
    • Continuous feedback
  • Affective Computing
    • Research from MIT
    • Various mechanisms to detect AND affect

Slide 26 (of 45)

research something real

R 18

Research  Something Real
  • Phew! A lot of research
  • Many topics not examined today
  • … What were we trying to do again?

Slide 27 (of 45)

hypothesis1
Hypothesis
  • Perceived emotional content of music can be influenced by controlling both the structural and performative aspects
  • Audience feedback can be captured by computers to tailor the musical experience

Slide 28 (of 45)

the rule system influence

Sy 1

The Rule System – Influence
  • Influencing perceived emotions
    • E.g. make “happier”
  • How?
  • Apply octant-grouped structural rules
  • E.g. “Influence to be upbeat and positive”
    • Apply octant 2 rules (tempo [fast], loudness [loud] …) to music structure

Slide 29 (of 45)

the rule system detection

Sy 2

The Rule System – Detection
  • Why?
  • Good influencing needs emotional context
  • Requires
    • Model of Music Tension
    • Advanced pattern matching
    • Advanced knowledge of music theory and composition
  • Very tricky … Not attempted before

Slide 30 (of 45)

the architecture

Sy 3

The Architecture

Slide 31 (of 45)

the architecture application intent

Sy 4

The Architecture – Application Intent
  • Emotive Information
    • The [Arousal, Valence] vector [3]

Slide 32 (of 45)

the architecture audience sensing

Sy 5

The Architecture – Audience Sensing
  • Audience response provides a wealth of feedback data to performer
  • Attitudes and audience response [A, V] incorporated
    • E.g. Cap the fearfulness of a room’s music
  • Affective computing
    • Keystroke/mouse movement: Arousal and Tension
    • Gaze tracking/Skin conductivity: Arousal and Interest
    • Same problems as measuring induced emotion though

Slide 33 (of 45)

the architecture emotive algorithm

Sy 6

The Architecture – Emotive Algorithm
  • Equalising unit for [A, V] coming from game and audience
  • Player Cap:
  • Room Value:
  • Game Event:
  • Resulting [A, V]:

Slide 34 (of 45)

testing progress 2

Sy 7

Testing Progress [2]
  • Aims
    • Influence the perceived emotions of music with primary music emotion structural rules
    • Rules can apply to both Western classical and standard computer game music
  • Testing
    • Listener played original work, followed by altered work (e.g. apply octant 2 rules)
    • How did emotion baseline change?
    • 11 participants, played 6 altered versions

Slide 35 (of 45)

testing progress

Sy 8

Testing Progress
  • Overall Results
  • Looks OK but why the A, V discrepancy?

Slide 36 (of 45)

testing progress1

Sy 9

Testing Progress
  • Quadrant Breakdown

Slide 37 (of 45)

testing progress2

Sy 10

Testing Progress
  • Not Angrier, why?
    • Music selection (something deeper going on ..?)
    • Incomplete rule implementation for quadrant 2

Slide 38 (of 45)

methodology recap
Methodology Recap
  • Score – Manipulate the structure and mark up with emotional performance metadata
  • How are we going?
    • Structure: Implemented and progressing
    • Performative: Identified, future implementation

Slide 39 (of 45)

methodology recap1
Methodology Recap
  • Audience – Determine emotional state and attitudes using affective computing tech.
  • How are we going?
    • Identified and developed a theoretical implementation

Slide 40 (of 45)

methodology recap2
Methodology Recap
  • Architecture – Bring together for an awareness of score and audience
  • How are we going?
    • Theoretical Implementation at present

Slide 41 (of 45)

future work

Fu

Future Work
  • Implement more structural music rules
  • Expanded testing regime
  • Implement performative rules
  • Begin testing of performative rules
  • Detection component
  • Incorporate Affective Computing elements

Slide 42 (of 45)

summary

Su 1

Summary
  • There exists no automated method to detect and influence the emotions of music
    • We’re getting there
  • An audience’s response to computer music has previously been inaccessible to computers and thus lost
    • Theoretical, still Future work

Slide 43 (of 45)

questions

Su 2

Questions?
  • Contact
    • srl@itee.uq.edu.au
    • http://itee.uq.edu.au/~srl
  • Papers
    • [1] "Playing with Affect: Music Performance with Awareness of Score and Audience", 2005, Australasian Computer Music Conference
    • [2] "Dynamic Response: Real-Time Adaptation for Music Emotion", 2005, Australasian Conference on Interactive Entertainment
    • [3] “Influencing the Perceived Emotions of Music with Intent”, 2005, Third International Conference on Generative Systems (in review)

Slide 44 (of 45)

references

Su 3

References
  • [a] Brown, S., B. Merker, and N.L. Wallin, An Introduction to Evolutionary Musicology, in The Origins of Music, S. Brown, B. Merker, and N.L. Wallin, Editors. 2000, MIT Press.
  • [b] Sloboda, J.A. and S.A. O'Neill, Emotions in Everyday Listening to Music, in Music and Emotion, theory and research. 2001, Oxford Press. p. 415-429.
  • [c] Russell, J.A., Measures of emotion., in Emotion: Theory research and experience., R.P.H. Kellerman, Editor. 1989, New York: Academic Press. p. 81-111.
  • [d] Schubert, E., Measurement and Time Series Analysis of Emotion in Music. 1999, University of New South Wales.
  • [e] Meyer, L.B., Emotion and Meaning in Music. 1956: The University of Chicago Press.

Slide 45 (of 45)