Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience

1. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press Chapter 7 Hearing and Speech “it was very fortunate that, even in Helmholtz’ time, the great anatomical discoveries by Corti (and others) had already made it clear that the vibrating tissue most important for hearing is the basilar membrane of the inner ear, because the cells on which the nerve endings terminate are seated on this membrane … the problem of how we hear was reduced largely to a mechanical question: how does the basilar membrane vibrate when the eardrum is exposed to a sinusoidal sound pressure?” Bekesy, Nobel Prize Lecture (online)

2. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press Chapter Outline 1.0 Introduction 2.0 The central auditory system 3.0 Functional mapping of auditory processing 4.0 Speech Perception 5.0 Music Perception 6.0 Learning and plasticity 7.0 Auditory awareness and imagery 8.0 Summary

3. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction A model for sound processing Sound inputs enter the system and there is a very brief storage or buffer called echoic memory. There may be many competing sounds occurring at the same time -- such as the sounds in a noisy lecture hall

4. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction A model for sound processing Selective attention allows the system to direct resources to a subset of the sensory inputs that are competing for attention -- such as attending to a lecturer and ignoring the sounds of someone whispering behind you

5. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction A model for sound processing Complex interactions occur between the new inputs and existing memories and experiences, as well with other sensory systems.

6. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction A model for sound processing The ultimate goal or ‘action’ to be performed is important as well, and will affect how information is encoded and stored. These processes are not one-way: there are interactions that occur through the encoding of sounds, both within the auditory system and across other sensory, cognitive, memory, and motor systems.

7. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction It’s about time … Time is a critical aspect of auditory processing: the auditory system differs from the visual system in that all sound processing occurs over time. A spectrogram provides a ‘picture’ of speech, with time on the x-axis and frequency on the y-axis. Speech contains harmonics -- formants - at specific regions on the spectrogram. Darker bands indicate more energy at that region in the speech signal. A spectrogram showing the energy at different frequencies in the speech signal

8. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction It’s about time … Three time scales are critical for decoding speech -- 20, 200, 2000 milliseconds (ms): Formants in consonants such as ‘b’ and ‘d’ change on a time scale of about 20/1000 of a second, or 20 ms A syllable lasts about 200 ms A typical sentence lasts around 2000 ms A spectrogram showing the energy at different frequencies in the speech signal

9. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction It’s about time … The auditory system must be able to decode information at each of these timescales in order to accurately perceive speech. A spectrogram showing the energy at different frequencies in the speech signal

10. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction Sound and hearing basics spectral-domain time-domain Two ways to represent the physical features of a sound pictorially: In the time-domain: time is on the x-axis and pressure (or amplitude) is on the y-axis In the spectral-domain: time is on the x-axis and frequency (in Hz) is on the y-axis

11. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction Sound and hearing basics Physical features of sounds: Frequency, intensity, and time The frequency of a sound is the rate of sound wave vibration and is calculated in terms of cycles completed per second, or hertz (Hz). A sinusoid with 1000 cycles per second has a frequency of 1000 Hz

12. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction Sound and hearing basics The intensity of a sound reflects the amount of amplitude (or displacement) within its cycle and over time. Intensity is reported in units of decibels (dB). Hearing thresholds for humans and cats are shown, along with the dB level for various types of sounds. Sounds over 120 dB present a high risk for damage to the hearing system.

13. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction Sound and hearing basics The peripheral hearing system: external, middle and inner ear The cochlea is located in the inner ear. Within the cochlea lies the basilar membrane

14. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction A cut-away view of the external, middle, and inner ear showing the transmission of sound (red arrows). Note the basilar membrane curving within the cochlea

15. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 1.0 Introduction Innervation patterns of afferent (ascending pathway) and efferent (descending pathway) neurons in the Organ of Corti

16. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System Auditory pathways The auditory system is complex, with many stages and pathways on the way to auditory cortex. The two main flow of information are in the ascending and descending pathways. The ascending (afferent) pathways transmit information about sounds from the periphery to cortex. This is not a simple delivery system, but entails many stages where information is encoded and recoded

17. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System Auditory pathways The descending (efferent) pathways extend from regions in the cortical and subcortical auditory system to the periphery. These pathways are under direct or indirect cortical control. An important function of the descending pathways is to provide ‘top-down’ information that aids in selective attention processes.

18. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System Auditory cortex Auditory cortex is the region within the cortex that is specialized for sound processing. It is located within the Sylvian fissure on the surface of the supratemporal plan and the upper banks of the superior temporal gyrus, so it is difficult to see from a lateral view of the brain. Brodmann areas for auditory and receptive language processing include Brodmann 22, 41, 42, and 52.

19. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System Auditory cortex Much of what we know about auditory cortex comes from studies in non-human primates. Here is a map of auditory regions in the left hemisphere of the macaque. The core, belt, and parabelt regions that form auditory cortex are tucked inside the Sylvian fissure and shown in an opened cutaway.

20. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System Auditory cortex Human brain, showing auditory cortical regions within the Sylvian fissure in the left hemisphere.

21. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System An axial slice of the brain showing a schematic of regions within auditory cortex. Note the hemispheric asymmetries for each region.

22. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System Auditory cortex Top: MRI coronal slice showing the location of the supratemporal plane Bottom: MRI axial slice showing the transverse gyrus of Heschl and the planum temporale

23. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 2.0 The Central Auditory System Receptive field properties of neurons in auditory cortex Here are receptive fields of two auditory cortical neurons plotted as a function of sound pressure level and azimuth (a ‘map’ of auditory space with respect to the two ears). Notice that the top map shows a cell that is broadly tuned across much of the azimuth on the contralateral side, while the bottom map shows a cell with a much narrower tuning.

24. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 3.0 Functional Mapping of Auditory Processing Auditory stream analysis A core function of the auditory system is to localize sound. Another central function is to recognize auditory ‘objects’ such as a friend’s voice or the sound of a car braking. An important role for the auditory system in humans is the perception of speech. These differing functions have led to the suggestion that there are multiple -- but highly interactive -- auditory processing ‘streams’ that underlie these functions and that interface with other sensory modalities.

25. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 3.0 Functional Mapping of Auditory Processing Heschl’s gyrus and the Planum temporale Heschl’s gyrus and the planum temporale have been the focus of many fMRI studies in order to understand their functional role in sound processing and speech perception. Heschl’s gyrus is the site for primary auditory cortex (A1). Far less is known about the functional role of the planum temporale. Recent studies have suggested that it serves as a nexus or hub for auditory processing streams.

26. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 3.0 Functional Mapping of Auditory Processing ‘What’ and ‘where’ processing streams ‘What’ and ‘where’ processing streams have been suggested based on studies in non-human primate. Similar to the pathways or streams evidenced in the visual system, the ‘what’ pathway is engaged in object recognition while the ‘where’ pathway is engaged in sound localization

27. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 3.0 Functional Mapping of Auditory Processing ‘What’ and ‘where’ processing streams Human studies have provided evidence that auditory streams may not simply form two pathways: this topic remains an open one, but recent studies have provided evidence for a ‘how’ stream in addition to the ‘what’ and ‘where’ streams. Other studies have provided evidence for a ‘who’ stream for voice recognition.

28. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 4.0 Speech Perception Speech perception The decoding of human speech is a central and important function of the human auditory system. Here is a spectrogram showing time on the x-axis, frequency on the y-axis, with darkened shadings showing increased energy at that time and frequency. Speech perceptual systems must be able to decode information in short time windows (20 ms) as well as longer ones (2000 ms).

29. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 4.0 Speech Perception Distinctive features Early work using the spectrogram to study the sound-based (acoustic phonetics) aspects of speech identified distinctive features of consonants and vowels. Here is a schematic illustration of the formant patterns for types of consonants in American English

30. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 4.0 Speech Perception The Lack of Invariance Problem Speech scientists were searching for the basic ‘building blocks’ or primitives that enabled us to decode speech. The distinctive features in speech sounds seemed like a promising candidate for speech primitives -- however the lack of invariant formant patterns for consonants, differing depending on the following vowel, made it obvious that encoding distinctive feature information could not be the only way speech decoded.

31. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 4.0 Speech Perception A process model Other models for understanding how speech perception was accomplished by the brain took into account other processes -- such as semantic decoding -- that were closely association with speech perception

32. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 4.0 Speech Perception Speech perception in the brain The advent of functional neuroimaging techniques allow us to investigate brain regions involved both in speech perception and production

33. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 4.0 Speech Perception 19th and 20th Century models for speech and language The groundbreaking work of 19th century scientists such as Broca and Wernicke identified regions that appeared critical for speech perception (temporal lobe, ‘Wernicke’s area’) and production (frontal lobe, ‘Broca’s area’). Functional imaging studies have confirmed these general findings, although new discoveries are still occurring to clarify the roles of these brain regions in speech.

34. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 4.0 Speech Perception Beyond Broca … A recent model extends the earlier views of receptive and productive language regions beyond Broca’s and Wernicke’s areas. This is an active area of study and new models are constantly being advanced.

35. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 5.0 Music Perception Music perception The perception of music is complex and likely involves many interactive stages

36. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 5.0 Music Perception Music perception Music and language are unique to human beings Recent studies have investigated if same or similar brain areas are involved in encoding music and language. Music perception in the brain is a relatively new field of study -- like speech, there are no simple ‘building blocks’ or primitives.

37. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 6.0 Learning and Plasticity Sensory memory formation We are continually learning to recognize new auditory ‘objects’ in our environment: a new friend’s voice, a new cellphone ringtone … how, and where, does the auditory system accomplish this? Sensory memory formation: a large body of evidence indicates that sensory memories are both formed and stored in auditory cortex.

38. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 7.0 Auditory Awareness and Imagery The hearing system is the last to fall asleep and the first to awaken. Why is it that the best way to wake someone up is to call their name? Is there a ‘wake-up’ area in the brain? A neuroimaging study investigated that question by playing simple sounds (beeps) and one’s own name in two conditions: while the subject was awake and while the subject was asleep

39. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 7.0 Auditory Awareness and Imagery A brain wake-up call? Results: areas in the middle temporal and frontal lobes were more active for hearing one’s name vs. beeps in the sleep condition. It might be that these regions monitor sound during sleep and activate other brain regions upon hearing one’s name.

40. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 7.0 Auditory Awareness and Imagery Are the same brain areas active when you imagine a sound vs. hearing it?

41. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 7.0 Auditory Awareness and Imagery An fMRI study investigated that question: primary auditory cortex (A1) was active during the perception phase but not during the imagery phase of the study.

42. Cognition, Brain and Consciousness: An Introduction to Cognitive Neuroscience Edited by Bernard J. Baars and Nicole M. Gage 2007 Academic Press 8.0 Summary • While much work has been accomplished to understand how the brain decodes complex sound patterns such as those in human speech and music, much more work needs to be done to understand how and where this happens in the brain • There are likely multiple auditory streams that interact in encoding auditory sensory inputs, that are also highly interactive with other sensory, memory, and action systems • The advent of neuroimaging techniques have made it possible to investigate aspects of auditory function that were previously difficult or impossible to study, such as auditory imagery and auditory awareness during sleep