slide1
Download
Skip this Video
Download Presentation
MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES

Loading in 2 Seconds...

play fullscreen
1 / 87

MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES - PowerPoint PPT Presentation


  • 326 Views
  • Uploaded on

MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES. Dominic W. Massaro. Old Fashioned View of Spoken Language and Communication. Anecdotal Evidence for Functional Value of Visible Speech. Persons with Hearing Loss Benjamin Franklin in France Hal in 2001: A Space Odyssey

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES' - PamelaLan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide2
MULTIMODAL EMOTION PERCEPTION:

ANALOGOUS TO SPEECH PROCESSES

Dominic W. Massaro

anecdotal evidence for functional value of visible speech
Anecdotal Evidence for Functional Value of Visible Speech
  • Persons with Hearing Loss
  • Benjamin Franklin in France
  • Hal in 2001: A Space Odyssey
  • “Hear TV Better with Glasses On”
  • Poorly Dubbed Foreign Films
value of talking heads
Value of Talking Heads
  • Enhance Realism and Naturalness
  • Convey Intention and Emotion
  • Enhance Intelligibility
research strategy to develop and evaluate the effectiveness of talking heads
Research Strategy to Develop and Evaluate the Effectiveness of Talking Heads
  • Auditory Synthetic speech
  • Computer Animated Talking Head
  • Development and Evaluation
psltalk baldi
PSLTalk (Baldi)
  • Successive Approximations to Realism
  • Real Time on PC Platforms
  • Controlled by text-to-speech synthesizer
slide12
Rotation of points
    • movement around axis, e.g., jaw rotation
slide13
Translation
    • movement of points, e.g., raise upper lip
slide14
Interpolation
    • Between two different subsections of wireframes--e.g., neck size
slide15
Scaling
    • constant multiplier, e.g., head width
features
Features
  • Driven by Text to Speech Engine
  • Target Values for each phoneme
  • Coarticulation
additional features
Additional Features
  • Coarticulation
  • Paralinguistic Properties
    • Nonspeech Segments
  • Texture Mapping
paralinguistic synthesis
Paralinguistic Synthesis
  • Nonspeech Segments
    • Breadth Noise, Cough, Clear Throat, Laugh, Lip Smack, Sneeze, Tongue Click, Burp

Baldi’snonspeech

psltalk baldi24
PSLTalk (Baldi)
  • Alignedwithnaturalauditory speech
evaluating intelligibility of baldi our talking head
Evaluating Intelligibility of Baldi, our Talking Head
  • Speechreading Syllables and Words
  • Understanding Sentences in Noise
  • Compare Baldi to Humans
additional features27
Additional Features
  • Tongue and Palate
  • Can Hide Noncritical Components
  • Can Reveal Normally-Hidden Parts
  • Can Highlight Interaction of Articulators
slide29
Language Training exercise

lay, ray, they

Top View

Side View

synthesis of emotion
Synthesis of Emotion
  • Voice is Informative
  • Face is More Critical
  • Basic Universal Emotions
    • Happiness, Anger, Surprise, Fear, Disgust, and Sadness, and Neutral
  • Can specify the degree to which Baldi expresses each of these emotions, and some combination
slide32
Can specify the degree to which Baldi expresses each of these emotions, and some combination

Over 80% correct in 6 alternative task

slide33
Emotion Training exercise

surprise

Front View

experimental strategy to study how emotion is processed
Experimental Strategy to Study How Emotion is Processed
  • Manipulate auditory and visual speech
  • present unimodal stimuli
  • present factorial bimodal stimuli
  • no feedback in task
  • test models of perception
expanded factorial design baldi
Expanded Factorial Design (Baldi)
  • Play Continuum-- voice fastest moving
  • Play Continuum-- face fastest moving
slide46
Brow

2

3

4

Happy

none

Angry

2

3

4

Happy

none

Angry

Mouth

expanded factorial design baldi47
Expanded Factorial Design (Baldi)
  • Play Continuum-- brow fastest moving
  • Play Continuum-- mouth fastest moving
fuzzy logical model of perception flmp
Fuzzy Logical Model of Perception (FLMP)
  • Continuous Information (Fuzzy Logic)
  • Independence of Sources
  • Multiplicative Integration of Sources
  • Optimal Integration Rule
prototypes of categories
Prototypes of Categories
  • Represent Ideal Properties
  • Fits Cognitive, Social Constructivists View
  • Categorization is Pattern Recognition
    • Formalization of Arnold’s Appraisal Method
example prototypes
Example Prototypes
  • Anger: Lowered Eyebrows ^ Lowered Lip Corners ^ Loud Voice ^ Angry Context ^ …
  • Happy: Raised Eyebrows ^ Raised Lip Corners ^ Long Duration ^ Happy Context ^ ...
slide55
A

i

Evaluation

V

j

a

v

i

j

Integration

s

k

Decision

R

k

slide57
A

HAP

2

3

4

ANG

none

HAP

2

3

4

ANG

none

V

.??

.7

.6

slide58
A

HAP

2

3

4

ANG

none

HAP

2

3

4

ANG

none

V

.78

.7

.6

slide59
FLMP
  • Evaluation: happy - a(happy) and v(happy)
  • angry - a(angry) and v(angry)
  • Integration: s(happy) = av
  • s(angry)= (1 - a)(1 - v)
  • Decision:
  • av
  • P(/da/) = --------------------------
  • av + (1 - a)(1 - v)
flmp example prediction
FLMP (example prediction)
  • Evaluation: auditory happy = .6
  • visual happy =.7
  • Integration: s(happy) = av = .6 x .7 = .42
  • s(angry) = (1 - a)(1 - v) = .4 x .3 = .12 Decision:
  • av .42
  • P(/hap/) = -------------------------- = ----------- = .78
  • av + (1 - a)(1 - v) .42 + .12
slide62
Ambiguity A = 1 - 2[|.5 - P(happy)|]

As P(happy) approaches 0 or 1,

A approaches 0.

Expect RT to increase with increases in ambiguity

slide64
Face

Angry 2 Happy None

Angry

2

Voice

Happy

None

expanded factorial design baldi65
Expanded Factorial Design (Baldi)
  • Play Continuum-- face fastest moving
  • Play Continuum-- voice fastest moving
pattern recognition
Pattern Recognition
  • Central to Cognition
  • Combine Many Sources of Information
  • Impose Meaning on Events
  • Universal Principle
slide75
Emotion

Attention Condition

slide76
Speech

Attention Condition

slide77
The perception of emotions by ear and by eye" by de Gelder and Vroomen. Cognition and Emotion, 2000, 14 (3), 313-320
slide85
Baldi’s Body

WithDom’s Face

ad