multisensory interplay n.
Skip this Video
Loading SlideShow in 5 Seconds..
Multisensory Interplay PowerPoint Presentation
Download Presentation
Multisensory Interplay

Loading in 2 Seconds...

play fullscreen
1 / 29

Multisensory Interplay - PowerPoint PPT Presentation

  • Uploaded on

Multisensory Interplay. Jon Driver and Toemme Noesselt. Sensory Research. Sensory Research. Behavioral Consequences of Multimodality. Joint estimates of single property Spatial Ventriloquism Auditory Driving McGurk Effect

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Multisensory Interplay' - ciara

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
multisensory interplay

Multisensory Interplay

Jon Driver and Toemme Noesselt

behavioral consequences of multimodality
Behavioral Consequences of Multimodality
  • Joint estimates of single property
    • Spatial Ventriloquism
    • Auditory Driving
    • McGurk Effect
  • Can modalities affect each other without creating a single unified percept?
    • Touch at a location can help perception of color
    • Sound-induced Illusory Flash
    • Orientation discrimination improves with multiple beeps
convergence zones
Convergence Zones
  • Superior Colliculus
    •  inputs from somatosensory, auditory, and visual areas
    •  Super or Sub-additive responses found for combination
additivity in sc
Additivity in SC
  • most likely with weak unisensory inputs
    • Ceiling effect?
    • Neural limitations?
  • Late onset in development
  • Depends on multisensory cortex
testing for convergence
Testing for Convergence
  • Anatomical studies:
    •  Direct connections between different sensory areas
  • Single-cell studies:
    • Response to stimulation from different modalities
  • Neuroimaging:
    • Large-scale responses based on BOLD signal
influences on sensory specific areas
Influences on 'Sensory-Specific' Areas
  • Growing body of research shows that sensory-specific areas might be an artifact of the studies done with them
  • Examine studies that use:
    • fMRI
    • EEG
    • Invasive recording in animal models
  • Caveat:
    • fMRI has been shown to respond to attention and imagery
  • For example, speech may be imagined when viewing lip movements 
fmri analysis
fMRI analysis
  • Inspired by Stein (SC), some look for sub-, super-additive responses
  • Maybe linearity is normal, though, so some use max or mean criteria
  • Difficult because of spatial resolution
convergence in v1
Convergence in V1?

Amedi, Jacobsen, Hendler, Malach, and Zohary, 2002

erp results example
ERP results example
  • Tactile stimulation
  • Visual cue
  • ERP extracted
erp studies
ERP studies
  • visual N1 enhanced when tactile stimulation occurred at same location as a visual event
  • visual P1 modified by task-irrelevant sound
  • P1 modified by attend-visual relative to attend-tactile conditions
erp studies1
ERP studies
  • ERPs show early multi-sensory effects (~30 ms)
  • Poor localization
  • Potential methodological confounds
invasive studies
Invasive Studies
  • Current-source densities (CSD) reflect local PSPs
  • Region of auditory association cortex
  • Location and timing of stimulation consistent with auditory feed-forward, visual feed-back 
invasive studies1
Invasive Studies
  • Posture may affect responses to auditory signals in A1
  • Tactile stimuli modulate initial response to auditory signals in A1
multisensory interplay1
Multisensory Interplay
  • Examples of converging zones of multi-sensory input
  • Examples of interplay: one modality affects another
  • What frameworks does this evidence suggest?
possible frameworks
Possible Frameworks
  • A) All Multisensory
  • B) Bimodal Brain Areas
  • C) Critical Feedback Circuitry
all multisensory
All Multisensory
  • Unlikely to be completely undifferentiated
  • Even primary sensory areas responsive to multiple modalities
all multisensory1
All Multisensory
  • Thalamus might be source of multisensory interplay
  • Tactile stimulation can affect first neural response in A1, hypothesized from thalamus
  • Found in gerbils, hard to study in humans
all multisensory2
All Multisensory
  • Direct coritco-cortical influences
  • Anatomical evidence: single synapse from AC-VC and AC-SSC, AC-OC
  • However, not as many as to conventional Multi-sensory areas
  • Role still unclear
all multisensory3
All Multisensory
  • Still overwhelmingly "sensory-specific"
bimodal brain areas
Bimodal Brain Areas
  • Less extreme version of account A
  • Similar to current account, with more multi-sensory regions
  • Parallel multi- and single-modality processing could explain early EEG modulation
bimodal brain areas1
Bimodal Brain Areas
  • Different areas in auditory areas may be connected to distinct visual areas
  • Bimodal interplay would be affected by transduction time, explaining BOLD response time differences 
feedback circuitry
Feedback Circuitry
  • Effects in primary areas might be feedback from convergence zones
  • Evidence from effective connectivity in fMRI, tactile stimulation increases visual response
  • Evidence from EEG source-localization: STS - VC
feedback circuitry1
Feedback Circuitry
  • Evidence from invasive recordings: late A1 stimulation from vision (speculation?)
  • Feedback can be tested directly, but very little has been done
  • Perhaps the rival frameworks are all valid for certain situations
  • Perhaps primary cortex responses in the blind and deaf can help tease out what the multisensory roles are
  • New techniques will allow testing causal interplay