Jaimie gilbert psychology 593 october 6 2005
Download
1 / 30

Jaimie Gilbert Psychology 593 October 6, 2005 - PowerPoint PPT Presentation


  • 101 Views
  • Uploaded on

Visual speech speeds up the neural processing of auditory speech van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) Proceedings of the National Academy of Sciences, 102(4), 1181-1186. Jaimie Gilbert Psychology 593 October 6, 2005. Audio-Visual Integration.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Jaimie Gilbert Psychology 593 October 6, 2005' - thimba


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Jaimie gilbert psychology 593 october 6 2005

Visual speech speeds up the neural processing of auditory speechvan Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) Proceedings of the National Academy of Sciences, 102(4), 1181-1186.

Jaimie Gilbert

Psychology 593

October 6, 2005


Audio visual integration
Audio-Visual Integration speech

  • Information from one modality (e.g., visual) can influence the perception of information presented in a different modality (e.g., auditory)

    • Speech in noise

    • McGurk Effect


Demonstration of mcgurk effect
Demonstration of McGurk Effect speech

Audiovisual Speech Web-Lab

http://www.faculty.ucr.edu/~rosenblu/lab-index.html

Arnt Maasø University of Oslo

http://www.media.uio.no/personer/arntm/McGurk_english.html


Unresolved questions about av integration
Unresolved questions about speechAV integration

  • Behavioral evidence exists for vision altering the perception of speech, but…

  • When does it occur in processing?

  • How does it occur?


Erps can help answer the when question
ERPs can help answer the “when” question speech

  • EEG/MEG studies have demonstrated AV integration effects using oddball/mismatch paradigms

    • These effects occur around 150-250 ms

  • A non-speech ERP study with non-ecologically valid stimuli demonstrated earlier interaction effects (40-95 ms) (Giard & Peronnet, 1999)

  • Does AV integration for speech occur earlier than 150-250 ms?


There s a debate about the how question
There’s a debate about the “how” question… speech

  • Enhancement

    • Audio-visual integration generates activity at multi-sensory integration sites, information possibly fed back to sensory cortices

  • VS.

  • Suppression

    • Reduction of stimulus uncertainty by two corresponding sensory stimuli reduces the amount of processing required


The experiments
The Experiments speech

  • 3 experiments were conducted

    • Each had behavioral and EEG measures

    • Behavioral: Forced choice task

    • EEG: Auditory P1/N1/P2

  • 26 participants

    • Experiment 1: 16

    • Experiment 2: 10

    • Experiment 3: 10 (of the 16 who participated in Experiment 1)


The stimuli

Audio /pa/ speech

Audio /ta/

Audio /ka/

Visual /pa/

Visual /ta/

Visual /ka/

AV /pa/

AV /ta/

AV /ka/

Incongruent AV with Audio /pa/ + Visual /ka/

1 Female face & voice for all stimuli

In Exp. 1 & 2, each stimuli presented 100 times; total of 1000 trials

The Stimuli


Experiment 1
Experiment 1 speech

  • Exp. 1

  • Stimuli presented in blocks of audio, or blocks of visual, or blocks of AV (congruent and incongruent)

  • Participants knew before each block which stimuli were going to be presented


Experiment 2
Experiment 2 speech

  • Exp. 2

  • Stimuli presented in randomized blocks containing all stimuli types (A, V, Congruent AV, Incongruent AV) to reduce expectancy

  • Task for both experiments: choose which stimuli was presented; for AV--choose what was heard while looking at the face


Experiment 3
Experiment 3 speech

  • Presented 200 Incongruent AV stimuli

  • Task: choose what syllable you saw, neglect what you heard

  • In all experiments, correct response to Incongruent AV = /ta/


Waveform analysis
Waveform Analysis speech

  • Retained 75-80% of recordings after Artifact Rejection and Ocular Artifact Reduction

  • Only correct responses were analyzed

  • 6 electrodes used in analysis: FC3, FC4, FCz, CPz, P7, P8

  • Reference electrodes: Linked mastoids


Results
Results speech

  • This study’s answer to “How”

    • Suppression/Deactivation Hypothesis

  • AV N1 & P2 amplitude were significantly reduced compared to Auditory-alone peaks

  • Performed separate analysis to determine if summing the responses to unimodal stimuli would result in the amplitude reduction present in the data—this was not the case; therefore the AV waveform is not a superposition of the 2 sensory waveforms, but reflects actual multisensory interaction.


Results experiment 1
Results: Experiment 1 speech

  • N1/P2 Amplitude

    • AV < A (p < .0001)

  • N1/P2 Latency

    • AV < A (significant, but confounded by interaction)

    • Modality x Stimulus Identity

      • P < T < K (p < .0001)

    • Latency effect more pronounced in P2, but can occur as early as N1


Results experiment 2
Results: Experiment 2 speech

  • N1/P2 Amplitude

    • AV < A (p < .0001)

  • N1/P2 Latency

    • AV < A (p < .0001)

    • Modality x Stimulus Identity (p < .06)


Results comparison of exp 1 exp 2
Results: comparison of Exp. 1 & Exp. 2 speech

  • Similar results for Exp. 1 & 2;

  • Temporal facilitation varied by Stimulus Identity but amplitude reduction did not;

  • No evidence for attention effect (i.e., for expectancy affecting waveform morphology)


Temporal facilitation depends on visual saliency signal redundancy
Temporal facilitation depends on visual saliency/signal redundancy

  • More temporal facilitation is expected to occur if:

    • The audio and the visual signals are redundant

    • The visual cue (which naturally precedes the auditory cue) is more salient

    • (Figure 3)


Results experiment 3 incongruent av stimuli
Results: Experiment 3/Incongruent AV Stimuli redundancy

  • Incongruent AV stimuli in Exp. 1 & 2:

    • no temporal facilitation

    • Amplitude reduction present and equivalent to reduction seen for Congruent AV stimuli

  • Experiment 3:

    • Both temporal facilitation and amplitude reduction occurred


Visual speech effects on auditory speech
Visual speech effects on auditory speech redundancy

  • Perceptual ambiguity/salience of visual speech affects processing time of auditory speech

  • Incorporating visual speech with auditory speech reduces the amplitude of N1/P2 “independent of AV congruency, participant’s expectancy, and attended modality” (p. 1184)


Ecologically valid stimuli
Ecologically valid stimuli redundancy

  • Suggest that AV speech processing is different from general multisensory integration due to the ecological validity of speech


Possible explanation for amplitude reduction
Possible explanation for amplitude reduction redundancy

  • Visemes provide information regarding place of articulation

  • If this information is salient and/or redundant with auditory place of articulation cues (e.g., 2nd and 3rd formants), the auditory cortex does not need to analyze these frequency regions, resulting in fewer firing neurons


Analysis by synthesis model of av speech perception
Analysis-by-Synthesis Model of AV Speech Perception redundancy

  • Visual speech activates internal representation/prediction

  • This representation/prediction is updated as more visual information is received over time

  • Representation/prediction is compared to the incoming auditory signal

  • Residual errors to this matching process are reflected by temporal facilitation and amplitude reduction effects

  • Attended modality can influence temporal facilitation


Suggest 2 time scales for av integration
Suggest 2 time scales for AV integration redundancy

  • 1: feature stage

    • 25 ms

    • Latency facilitation

    • (sub-)segmental analysis

  • 2: perceptual unit stage

    • 200 ms

    • Amplitude reduction

    • Syllable level analysis

    • Independent of feature content and attended modality


Summary
Summary redundancy

  • AV speech interaction occurs by the time N1 is elicited (50-100 ms)

  • Processing time of auditory speech varies by the saliency/ambiguity of visual speech

  • Amplitude of AV ERP reduced when compared to amplitude of A-alone ERP


Questions
Questions redundancy

  • Dynamic visual stimulus and ocular artifact

  • If effects of AV integration are influenced by attended modality, would modality dominance also influence these effects?

  • Are incongruent AV/McGurk stimuli ecologically valid?


ad