Brain, Psycholinguistics, & Cognitive Science

1. Brain, Psycholinguistics, & Cognitive Science

2. Outline • How does psycholinguistics fit within the umbrella of cognitive science? • What do we know about language and the brain?

3. Inter-relationships

4. What do cognitive psychologists do? Construct theories that describe how mental processing works in real time: • Within some domain, what are the mental representations and how are they manipulated? • How does information flow through the mind? • How is processing impacted by memory constraints, stimulus quality, mode of input, tasks, etc? Levels of processing Serial vs. parallel processing Top-down/bottom-up Automatic vs. strategic processing

5. What do (formal theoretical) linguists do? • Construct formal theories of our linguistic knowledge • Sets of rules/principles/assumptions for generating utterances • Criteria for a good theory • The rules generate all and only grammatical outputs (intuitions) • In the simplest way

6. Derivational Theory of Complexity • Miller’s clause-processing model of syntactic processing, motivated by Chomsky’s Transformational Grammar. • Example of taking a linguistic theory and trying to implement it directly as a psychological theory of processing.

7. Transformational Grammar • Two levels of syntactic structure, related by transformational rules; accounts for similarity in meaning across different strings. • Surface structure ((John) (picked __ (the box) up)). • Transformations Particle Movement • Deep Structure ((John) (picked up (the box))). • Phrase structure rules and lexical insertion rules used to construct DS

8. Diff SS, Same DS Transformations can move, delete, or add words. Complex SS’s require a sequence of transformations • Wh-movement & Subj/Aux Inversion SS: Which book did Jim buy __? SS Jim did buy a book. DS: Jim did buy a/which book • Passivization SS: The beer was drunk by Jim. SS: Jim drank the beer. DS: Jim drank the beer.

9. DTC • Determine the surface structure of the sentence • Reverse the transformations one by one • Recover the deep structure • Map DS to meaning

10. Processing Unit is entire Sentence • Wh-movement & Subj/Aux Inversion SS: Which book did Jim buy __? SS: Jim did buy a book. DS: Jim did buy a/which book • Passivization SS: The beer was drunk by Jim. SS: Jim drunk the beer. DS: Jim drunk the beer.

11. How would you test the DTC? • Early evidence seemed to support it. • Later evidence was problematic • No longer a viable theory of sentence comprehension.

12. Memory & Language • Semantic Memory: LTM storehouse of conceptual knowledge • What is a cow? What is truth? • Lexicon: LTM storehouse of knowledge about words • Lexical Entry: For each word/morpheme, Spelling, pronunciation, syntactic category, pointer to semantic memory • Episodic Memory: LTM storehouse of our experiences

13. uses is a is Spreading Activation Model (Collins & Loftus, 1975) • Length codes typicality • Link codes type of relation • Some redundancy

14. Commmon Current Assumptions about Semantic Memory & the Lexicon • Semantic Memory is a network of concepts, organized by semantic similarity • Lexicon is a network of words, organized by phonological similarity • Interconnections link meanings to words

15. Cognitive Neuroscience • Cognitive psychologists tend to talk about the architecture of the mind in terms of functionality • E.g., what is the input to word recognition? What is the output? • Ultimately, the mental operations described by cognitive psychology occur in the brain • In some cases, neuroscience can inform cognitive psychology • E.g., we may be able to learn about how words/concepts are represented by investigating activation patterns in response to different classes of words: action verbs activate motor cortex; perception verbs activate visual cortex

16. Brain Anatomy (& Language) • Is language localized in the brain? • Is language lateralized?

17. LH lobes Parietal lobe Frontal lobe Occipital lobe Temporal lobe Cerebellum

18. Some Lg-relevant areas

19. Video Clip (15 min) Brain Story: First among Equals • The first segment has been comparing human and chimp abilities to plan. They conclude that chimps can make and execute plans, but not as far into the future as humans. • We’ll watch a segment on LG • Aphasia patient • Mapping the brain prior to surgery

20. What did you learn from the clip? • What does it mean to have aphasia? • Do aphasics recover language function? How? • What is the current view on the role of Broca’s and Wernicke’s areas?

21. Schiff et al.(2005) • Do minimally conscious patients process speech?

22. Averaged healthy data Figure 1. Functional maps obtained during listening to narratives Yellow= forward; Blue = backward Red = both Schiff, N. D. et al. Neurology 2005;64:514-523

23. Figure 2. Volumes of activation during the passive listening tasks. The 2 patients are in blue and red; Averaged healthy activation in black. Schiff, N. D. et al. Neurology 2005;64:514-523

24. Schiff et al. • paper was published shortly after the Terry Schiavo media/political frenzy. • How do we decide whether a minimally conscious patient is experiencing a life worth living? • Is language comprehension relevant? • Why don’t we care as much about how their brain responds to smells?

25. Speech & Spoken Word Recognition

26. Outline • Why is speech perception difficult for computers to do? • Problem of Invariance • How do humans do it so easily? • Bottom-up information (acoustic signal) • Top-down information (higher level context)

27. What I said bookmark it Motorolla modem port a procedure and then stick it in the mail movie clips I might add Inscrutable the right or left What was transcribed book market motor roll a mode import upper seizure and dense thicket in the mail move eclipse I my dad in screw double the writer left Same sequences of phonemes form different words Different phonemes, at underlying level Automatic Speech Recognition Follies (David Pogue, NYTimes, 8/15/02)

28. The Problem of Invariance • Individual phonemes do not have invariant acoustic cues. • There’s a lot of variability in the acoustic signal! • Variance in the acoustic signal has many sources: • Coarticulation • Differences among speakers • Differences within speakers: yelling/whispering, phone/in-person, etc. • If the bottom-up acoustic signal doesn’t provide consistent cues, how do we recognize phonemes/words?

29. Reading a spectrogram formants Can you see invariant cues associated with /i/ ?

30. If we can’t/don’t rely solely on bottom-up input, how do we recognize speech? • Perceive speech as (intended?) articulatory gestures, not as acoustic signal? • McGurk Effect • Sine Wave Speech: There are no essential acoustic properties that enable speech perception. Rather second-order changes in frequency and amplitude over time are responsible. sine wave speech demos • Use top-down information (word and sentence context) to complement bottom-up information • If so, when and how?

31. Is Speech Special? Modularity Thesis (Fodor, 1983) • The mind is not a unified whole. In addition to central processes, there are specialized input-output modules • Central: decision making • Input: color perception, voice recognition • Output: throwing, touch-typing, articulation • Modules are fast, informationally-encapsulated, mandatory, exhibit characteristic breakdowns, and have shallow outputs. • Speech perception may be handled by a specialized input module

32. The Essence of Motor Theory • Speech perception is grounded in our knowledge of speech production. We recognize phonemes by covertly re-creating the articulatory gestures. (Lieberman et al., 1967) • Consistent with philosophy that performance & perception are inextricably linked. • Assume innate, encapsulated phonetic module

33. Spoken Word Recognition • Overcoming the problem of invariance in speech perception • Motor Theory • Top-Down Feedback: Word to Phoneme • TRACE • Cohort theory of Spoken Word Recognition

34. Evidence for Top-Down influence on speech perception • Phoneme Restoration Effect (Warren, 1970) • Lexical bias in categorical perception task, e.g. dype vs. type (Clifton & Connine, 1987)

35. TRACE(McClelland & Elman, 1986) • Interactive connectionist model • Nodes in network represent phonetic features, segments, & words • Feature nodes activated by consistent input • Activation spreads up through network & back down again • Predicts top-down effects

36. Example: initial phoneme in “pick” is ambiguous betwn /b/ & /p/.(Lexical Bias)

37. Is word recognition Automatic & Modular? Automatic Processes • Fast • Do not require attention • Feed-forward (can’t be guided, controlled, or stopped midstream) • Not subject to top-down feedback (informational encapsulation)

38. Stroop Effect Name font color RED GREEN BLUE YELLOW GREEN What happens if you have to name word?

39. Differences between spoken and written word recognition • For relatively short words, letters in a written word are processed in parallel • Eye movement data • Word superiority effect • Letter-Search Task • Spoken word unfolds across time • Can recognize some words before they are completely pronounced.

40. Research on the Lexicon (Outline) • How are lexical entries accessed? (Word Recognition) What is the input? • Speaking (Ashcraft) • Reading • Listening • How is lexical ambiguity resolved?