Download
1 / 87

Anecdotal Evidence for Functional Value of Visible Speech - PowerPoint PPT Presentation


  • 326 Views
  • Uploaded on

MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES. Dominic W. Massaro. Old Fashioned View of Spoken Language and Communication. Anecdotal Evidence for Functional Value of Visible Speech. Persons with Hearing Loss Benjamin Franklin in France Hal in 2001: A Space Odyssey

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Anecdotal Evidence for Functional Value of Visible Speech' - PamelaLan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide2 l.jpg

MULTIMODAL EMOTION PERCEPTION:

ANALOGOUS TO SPEECH PROCESSES

Dominic W. Massaro



Anecdotal evidence for functional value of visible speech l.jpg
Anecdotal Evidence for Functional Value of Visible Speech

  • Persons with Hearing Loss

  • Benjamin Franklin in France

  • Hal in 2001: A Space Odyssey

  • “Hear TV Better with Glasses On”

  • Poorly Dubbed Foreign Films



Value of talking heads l.jpg
Value of Talking Heads

  • Enhance Realism and Naturalness

  • Convey Intention and Emotion

  • Enhance Intelligibility


Research strategy to develop and evaluate the effectiveness of talking heads l.jpg
Research Strategy to Develop and Evaluate the Effectiveness of Talking Heads

  • Auditory Synthetic speech

  • Computer Animated Talking Head

  • Development and Evaluation


Psltalk baldi l.jpg
PSLTalk (Baldi) of Talking Heads

  • Successive Approximations to Realism

  • Real Time on PC Platforms

  • Controlled by text-to-speech synthesizer


Slide11 l.jpg

Baldi of Talking HeadsDescribesHimself


Slide12 l.jpg


Slide13 l.jpg

  • Translation of Talking Heads

    • movement of points, e.g., raise upper lip


Slide14 l.jpg

  • Interpolation of Talking Heads

    • Between two different subsections of wireframes--e.g., neck size


Slide15 l.jpg

  • Scaling of Talking Heads

    • constant multiplier, e.g., head width


Features l.jpg
Features of Talking Heads

  • Driven by Text to Speech Engine

  • Target Values for each phoneme

  • Coarticulation


Additional features l.jpg
Additional Features of Talking Heads

  • Coarticulation

  • Paralinguistic Properties

    • Nonspeech Segments

  • Texture Mapping


Paralinguistic synthesis l.jpg
Paralinguistic Synthesis of Talking Heads

  • Nonspeech Segments

    • Breadth Noise, Cough, Clear Throat, Laugh, Lip Smack, Sneeze, Tongue Click, Burp

Baldi’snonspeech


Slide22 l.jpg

Texture Mapping of Talking Heads


Psltalk baldi24 l.jpg
PSLTalk (Baldi) of Talking Heads

  • Alignedwithnaturalauditory speech


Evaluating intelligibility of baldi our talking head l.jpg
Evaluating Intelligibility of Baldi, our Talking Head of Talking Heads

  • Speechreading Syllables and Words

  • Understanding Sentences in Noise

  • Compare Baldi to Humans


Additional features27 l.jpg
Additional Features of Talking Heads

  • Tongue and Palate

  • Can Hide Noncritical Components

  • Can Reveal Normally-Hidden Parts

  • Can Highlight Interaction of Articulators



Slide29 l.jpg

Language Training exercise of Talking Heads

lay, ray, they

Top View

Side View


Slide30 l.jpg

Language Training exercise of Talking Heads

wah, rah

Back View


Synthesis of emotion l.jpg
Synthesis of Emotion of Talking Heads

  • Voice is Informative

  • Face is More Critical

  • Basic Universal Emotions

    • Happiness, Anger, Surprise, Fear, Disgust, and Sadness, and Neutral

  • Can specify the degree to which Baldi expresses each of these emotions, and some combination


Slide32 l.jpg

Can specify the degree to which Baldi expresses each of these emotions, and some combination

Over 80% correct in 6 alternative task


Slide33 l.jpg

Emotion Training exercise these emotions, and some combination

surprise

Front View


Emotion hallucinations l.jpg
Emotion Hallucinations these emotions, and some combination


Speech hallucinations l.jpg
Speech Hallucinations these emotions, and some combination


Experimental strategy to study how emotion is processed l.jpg
Experimental Strategy to Study How Emotion is Processed these emotions, and some combination

  • Manipulate auditory and visual speech

  • present unimodal stimuli

  • present factorial bimodal stimuli

  • no feedback in task

  • test models of perception


Expanded factorial design baldi l.jpg
Expanded Factorial Design (Baldi) these emotions, and some combination

  • Play Continuum-- voice fastest moving

  • Play Continuum-- face fastest moving


Slide44 l.jpg

Two Sources Better than either One Alone these emotions, and some combination


Slide45 l.jpg

Unique Response from Two Sources these emotions, and some combination


Slide46 l.jpg

Brow these emotions, and some combination

2

3

4

Happy

none

Angry

2

3

4

Happy

none

Angry

Mouth


Expanded factorial design baldi47 l.jpg
Expanded Factorial Design (Baldi) these emotions, and some combination

  • Play Continuum-- brow fastest moving

  • Play Continuum-- mouth fastest moving


Slide48 l.jpg

Identification Judgments these emotions, and some combination


Slide49 l.jpg

Rating Judgments these emotions, and some combination


Slide51 l.jpg

Fuzzy Logic these emotions, and some combination


Fuzzy logical model of perception flmp l.jpg
Fuzzy Logical Model of Perception (FLMP) these emotions, and some combination

  • Continuous Information (Fuzzy Logic)

  • Independence of Sources

  • Multiplicative Integration of Sources

  • Optimal Integration Rule


Prototypes of categories l.jpg
Prototypes of Categories these emotions, and some combination

  • Represent Ideal Properties

  • Fits Cognitive, Social Constructivists View

  • Categorization is Pattern Recognition

    • Formalization of Arnold’s Appraisal Method


Example prototypes l.jpg
Example Prototypes these emotions, and some combination

  • Anger: Lowered Eyebrows ^ Lowered Lip Corners ^ Loud Voice ^ Angry Context ^ …

  • Happy: Raised Eyebrows ^ Raised Lip Corners ^ Long Duration ^ Happy Context ^ ...


Slide55 l.jpg

A these emotions, and some combination

i

Evaluation

V

j

a

v

i

j

Integration

s

k

Decision

R

k


Slide57 l.jpg

A these emotions, and some combination

HAP

2

3

4

ANG

none

HAP

2

3

4

ANG

none

V

.??

.7

.6


Slide58 l.jpg

A these emotions, and some combination

HAP

2

3

4

ANG

none

HAP

2

3

4

ANG

none

V

.78

.7

.6


Slide59 l.jpg
FLMP these emotions, and some combination

  • Evaluation: happy - a(happy) and v(happy)

  • angry - a(angry) and v(angry)

  • Integration: s(happy) = av

  • s(angry)= (1 - a)(1 - v)

  • Decision:

  • av

  • P(/da/) = --------------------------

  • av + (1 - a)(1 - v)


Flmp example prediction l.jpg
FLMP (example prediction) these emotions, and some combination

  • Evaluation: auditory happy = .6

  • visual happy =.7

  • Integration: s(happy) = av = .6 x .7 = .42

  • s(angry) = (1 - a)(1 - v) = .4 x .3 = .12 Decision:

  • av .42

  • P(/hap/) = -------------------------- = ----------- = .78

  • av + (1 - a)(1 - v) .42 + .12


Slide62 l.jpg

Ambiguity A = 1 - 2[|.5 - P(happy)|] these emotions, and some combination

As P(happy) approaches 0 or 1,

A approaches 0.

Expect RT to increase with increases in ambiguity


Slide64 l.jpg

Face these emotions, and some combination

Angry 2 Happy None

Angry

2

Voice

Happy

None


Expanded factorial design baldi65 l.jpg
Expanded Factorial Design (Baldi) these emotions, and some combination

  • Play Continuum-- face fastest moving

  • Play Continuum-- voice fastest moving


Pattern recognition l.jpg
Pattern Recognition these emotions, and some combination

  • Central to Cognition

  • Combine Many Sources of Information

  • Impose Meaning on Events

  • Universal Principle


Hallucinations l.jpg
Hallucinations these emotions, and some combination


Slide75 l.jpg

Emotion these emotions, and some combination

Attention Condition


Slide76 l.jpg

Speech these emotions, and some combination

Attention Condition


Slide77 l.jpg

The perception of emotions by ear and by eye" by de Gelder and Vroomen. Cognition and Emotion, 2000, 14 (3), 313-320



Slide79 l.jpg

Identify Face and Vroomen.


Slide80 l.jpg

Identify Voice and Vroomen.



Slide83 l.jpg

Creating Conversations: and Vroomen. Gui for Text Markup Language


Slide84 l.jpg

Some Marked Up Text and Vroomen.


Slide85 l.jpg

Baldi’s and Vroomen. Body

WithDom’s Face


Slide86 l.jpg

Meet and Vroomen. Betty


ad