1 / 29

Multisensory Interplay

Multisensory Interplay. Jon Driver and Toemme Noesselt. Sensory Research. Sensory Research. Behavioral Consequences of Multimodality. Joint estimates of single property Spatial Ventriloquism Auditory Driving McGurk Effect

ciara
Download Presentation

Multisensory Interplay

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multisensory Interplay Jon Driver and Toemme Noesselt

  2. Sensory Research

  3. Sensory Research

  4. Behavioral Consequences of Multimodality • Joint estimates of single property • Spatial Ventriloquism • Auditory Driving • McGurk Effect • Can modalities affect each other without creating a single unified percept? • Touch at a location can help perception of color • Sound-induced Illusory Flash • Orientation discrimination improves with multiple beeps

  5. Convergence Zones • Superior Colliculus •  inputs from somatosensory, auditory, and visual areas •  Super or Sub-additive responses found for combination

  6. Additivity in SC • most likely with weak unisensory inputs • Ceiling effect? • Neural limitations? • Late onset in development • Depends on multisensory cortex

  7. Convergence Zones

  8. Testing for Convergence • Anatomical studies: •  Direct connections between different sensory areas • Single-cell studies: • Response to stimulation from different modalities • Neuroimaging: • Large-scale responses based on BOLD signal

  9. Influences on 'Sensory-Specific' Areas • Growing body of research shows that sensory-specific areas might be an artifact of the studies done with them • Examine studies that use: • fMRI • EEG • Invasive recording in animal models

  10. fMRI • Caveat: • fMRI has been shown to respond to attention and imagery • For example, speech may be imagined when viewing lip movements 

  11. fMRI analysis • Inspired by Stein (SC), some look for sub-, super-additive responses • Maybe linearity is normal, though, so some use max or mean criteria • Difficult because of spatial resolution

  12. Convergence in V1? Amedi, Jacobsen, Hendler, Malach, and Zohary, 2002

  13. ERP results example • Tactile stimulation • Visual cue • ERP extracted

  14. ERP studies • visual N1 enhanced when tactile stimulation occurred at same location as a visual event • visual P1 modified by task-irrelevant sound • P1 modified by attend-visual relative to attend-tactile conditions

  15. ERP studies • ERPs show early multi-sensory effects (~30 ms) • Poor localization • Potential methodological confounds

  16. Invasive Studies • Current-source densities (CSD) reflect local PSPs • Region of auditory association cortex • Location and timing of stimulation consistent with auditory feed-forward, visual feed-back 

  17. Invasive Studies • Posture may affect responses to auditory signals in A1 • Tactile stimuli modulate initial response to auditory signals in A1

  18. Multisensory Interplay • Examples of converging zones of multi-sensory input • Examples of interplay: one modality affects another • What frameworks does this evidence suggest?

  19. Possible Frameworks • A) All Multisensory • B) Bimodal Brain Areas • C) Critical Feedback Circuitry

  20. All Multisensory • Unlikely to be completely undifferentiated • Even primary sensory areas responsive to multiple modalities

  21. All Multisensory • Thalamus might be source of multisensory interplay • Tactile stimulation can affect first neural response in A1, hypothesized from thalamus • Found in gerbils, hard to study in humans

  22. All Multisensory • Direct coritco-cortical influences • Anatomical evidence: single synapse from AC-VC and AC-SSC, AC-OC • However, not as many as to conventional Multi-sensory areas • Role still unclear

  23. All Multisensory • Still overwhelmingly "sensory-specific"

  24. Bimodal Brain Areas • Less extreme version of account A • Similar to current account, with more multi-sensory regions • Parallel multi- and single-modality processing could explain early EEG modulation

  25. Bimodal Brain Areas • Different areas in auditory areas may be connected to distinct visual areas • Bimodal interplay would be affected by transduction time, explaining BOLD response time differences 

  26. Feedback Circuitry • Effects in primary areas might be feedback from convergence zones • Evidence from effective connectivity in fMRI, tactile stimulation increases visual response • Evidence from EEG source-localization: STS - VC

  27. Feedback Circuitry • Evidence from invasive recordings: late A1 stimulation from vision (speculation?) • Feedback can be tested directly, but very little has been done

  28. Remarks • Perhaps the rival frameworks are all valid for certain situations • Perhaps primary cortex responses in the blind and deaf can help tease out what the multisensory roles are • New techniques will allow testing causal interplay

More Related