Continuous detail is used in language comprehension and language learning:

1. Continuous detail is used in language comprehension and language learning: Temporal Integration at two time scales Bob McMurray University of Iowa Dept. of Psychology

2. Collaborators Richard Aslin Michael Tanenhaus David Gow Joe Toscano Cheyenne Munson Meghan Clayards Dana Subik Julie Markant The students of the MACLab

3. Temporal Integration Temporal integration: a critical problem for cognition. - information never arrives synchronously. • Vision: integration across head-movements, saccades and attention-shifts. • Music perception:long-term dependencies and short term expectancies.

4. In language, information arrives sequentially. • Partial syntactic and semantic representations are formed as words arrive. The Hawkeyes beat the Illini (once) • Words are identified over sequential phonemes.

5. Spoken Word Recognitionis an ideal arena in which to study these issues because: • Speech production gives us a lot of rich temporal information to use in this way. • We have a clear understanding of the input (from phonetics). • The output is easy to measure online (visual world paradigm).

6. Scales of temporal integration in word recognition • A Word: ordered series of articulations. • - Build abstract representations. • - Form expectations about future events. • - Fast (online) processing. • A phonology: • - Abstract across utterances. • - Expectations about possible future events. • - Slow (developmental) processing

7. Mechanisms of Temporal Integration • Stimuli do not change arbitrarily. • Perceptual cues reveal something about the change itself. • Active integration: • Anticipating future events • Retain partial present representations. • Resolve prior ambiguity.

8. Overview • Speech perception and Spoken Word Recognition. 2) Lexical activation is sensitive to fine-grained detail in speech. 3) Fast temporal integration: taking advantage of regularity in the signal for temporal integration. 4) Slow temporal integration: Developmental consequences

9. X basic bakery bakery X ba… kery barrier X X bait barricade X baby • Online Word Recognition • Information arrives sequentially • At early points in time, signal is temporarily ambiguous. • Later arriving information disambiguates the word.

10. Current models of spoken word recognition • Immediacy: Hypotheses formed from the earliest moments of input. • Activation Based: Lexical candidates (words) receive activation to the degree they match the input. • Parallel Processing: Multiple items are active in parallel. • Competition: Items compete with each other for recognition.

11. Input: b... u… tt… e… r time beach butter bump putter dog

12. These processes have been well defined for a phonemic representation of the input. k A g n I S  n But considerably less ambiguity if we consider subphonemic information. Example: subphonemic effects of motor processes.

13. Coarticulation n n ee t c k Any action reflects future actions as it unfolds. Example:Coarticulation Articulation (lips, tongue…) reflectscurrent, futureandpastevents. Subtle subphonemic variation in speech reflects temporal organization. Sensitivity to theseperceptualdetails might yield earlier disambiguation.

14. These processes have largely been ignored because of a history of evidence that perceptual variability gets discarded. Example:Categorical Perception

15. Categorical Perception B 100 100 Discrimination % /p/ Discrimination ID (%/pa/) 0 0 B VOT P • Sharp identification of tokens on a continuum. P • Discrimination poor within a phonetic category. Subphonemic variation in VOT is discarded in favor of adiscretesymbol (phoneme).

16. Evidence against the strong form of Categorical Perception from psychophysical-type tasks: • Discrimination Tasks • Pisoni and Tash (1974) • Pisoni & Lazarus (1974) • Carney, Widin & Viemeister (1977) • Training • Samuel (1977) • Pisoni, Aslin, Perey & Hennessy (1982) • Goodness Ratings • Miller (1997) • Massaro & Cohen (1983)

17. Experiment 1 ? Does within-category acoustic detail systematically affect higher level language? Is there a gradient effect of subphonemic detail on lexical activation?

18. McMurray, Aslin & Tanenhaus (2002) A gradient relationshipwould yield systematic effects of subphonemic information on lexical activation. If this gradiency is useful for temporal integration, it must be preserved over time. Need a design sensitive to bothacoustic detailand detailedtemporal dynamicsof lexical activation.

19. Acoustic Detail Use a speech continuum—more steps yields a better picture acoustic mapping. KlattWorks:generate synthetic continua from natural speech. • 9-step VOT continua (0-40 ms) • 6 pairs of words. • beach/peach bale/pale bear/pear • bump/pump bomb/palm butter/putter • 6 fillers. • lamp leg lock ladder lip leaf • shark shell shoe ship sheep shirt

21. Temporal Dynamics How do we tap on-line recognition? With an on-line task:Eye-movements Subjects hear spoken language and manipulate objects in a visual world. Visual world includes set of objects with interesting linguistic properties. abeach, apeachand some unrelated items. Eye-movements to each object are monitored throughout the task. Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy, 1995

22. Why use eye-movements and visual world paradigm? • Relatively naturaltask. • Eye-movements generated veryfast(within 200ms of first bit of information). • Eye movementstime-lockedto speech. • Subjectsaren’t awareof eye-movements. • Fixation probability maps ontolexical activation..

23. Task A moment to view the items

25. Task Bear Repeat 1080 times

26. Identification Results 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 5 10 15 20 25 30 35 40 High agreement across subjects and items for category boundary. proportion /p/ B VOT (ms) P By subject:17.25 +/- 1.33ms By item: 17.24 +/- 1.24ms

27. Task 200 ms Trials 1 2 3 4 5 % fixations Time Target = Bear Competitor = Pear Unrelated = Lamp, Ship

28. Task 0.9 VOT=0 Response= VOT=40 Response= 0.8 0.7 0.6 0.5 Fixation proportion 0.4 0.3 0.2 0.1 0 0 400 800 1200 1600 2000 0 400 800 1200 1600 Time (ms) More looks to competitor than unrelated items.

29. Task target Fixation proportion Fixation proportion time time • Given that • the subject heard bear • clicked on “bear”… How often was the subject looking at the “pear”? Categorical Results Gradient Effect target target competitor competitor competitor competitor

30. Results 20 ms 25 ms 30 ms 10 ms 15 ms 35 ms 40 ms 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 0 400 800 1200 1600 0 400 800 1200 1600 2000 Response= Response= VOT VOT 0 ms 5 ms Competitor Fixations Time since word onset (ms) Long-lasting gradient effect: seen throughout the timecourse of processing.

31. 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0 5 10 15 20 25 30 35 40 Area under the curve: Clear effects of VOT B: p=.017* P: p<.001*** Linear Trend B: p=.023* P: p=.002*** Response= Response= Looks to Competitor Fixations Looks to Category Boundary VOT (ms)

32. 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0 5 10 15 20 25 30 35 40 Unambiguous Stimuli Only Clear effects of VOT B: p=.014* P: p=.001*** Linear Trend B: p=.009** P: p=.007** Response= Response= Looks to Competitor Fixations Looks to Category Boundary VOT (ms)

33. Summary Subphonemic acoustic differences in VOT have gradient effect on lexical activation. • Gradient effect of VOT on looks to the competitor. • Effect holds even for unambiguous stimuli. • Seems to be long-lasting. Consistent with growing body of work using priming (Andruski, Blumstein & Burton, 1994; Utman, Blumstein & Burton, 2000; Gow, 2001, 2002).

34. Sensitivity & Use Word recognition is systematically sensitive to subphonemic acoustic detail. 2) Acoustic detail is represented as gradations in activation across the lexicon. This sensitivity enables the system to take advantage of subphonemic regularities for temporal integration. 4) This has fundamental consequences for development: learning phonological organization.

35. Lexical Sensitivity Word recognition is systematically sensitive to subphonemic acoustic detail. • Voicing • Laterality, Manner, Place • Natural Speech • Vowel Quality

36. Lexical Sensitivity Word recognition is systematically sensitive to subphonemic acoustic detail. • Voicing • Laterality, Manner, Place • Natural Speech • Vowel Quality ? Non minimal pairs • ? Duration of effect • (experiment 1)

37. bump pump dump bun bumper bomb 2) Acoustic detail is represented as gradations in activation across the lexicon (a lexical basis vector for speech). Input: b... u… m… p… time

38. Temporal Integration This sensitivity enables the system to take advantage of subphonemic regularities for temporal integration. • Regressive ambiguity resolution (exp 1): • Ambiguity retained until more information arrives. • Progressive expectation building (exp 2): • Phonetic distinctions are spread over time • Anticipate upcoming material.

39. Development 4) Consequences for development: learning phonological organization. • Learning a language: • Integrating input across many utterances to build long-term representation. • Sensitivity to subphonemic detail (exp 4 & 5). • Allows statistical learning of categories (model). To assimilation

40. Experiment 2 How long are gradient effects of within-category detail maintained? Can subphonemic variation play a role in ambiguity resolution? How is information at multiple levels integrated? ? ?

41. Misperception What if initial portion of a stimulus wasmisperceived? • Competitor still active • - easy to activate it rest of the way. • Competitor completely inactive • - system will “garden-path”. • P ( misperception )  distance from boundary. • Gradient activation allows the system to hedge its bets.

42. Gradient Sensitivity parakeet barricade / beIrəkeId/ vs. / peIrəkit/ barricade vs. parakeet Input: p/b eIr ə k i t… time Categorical Lexicon parakeet barricade

43. Methods 10 Pairs of b/p items.

44. X

45. Eye Movement Results 0 5 10 15 20 25 30 35 Barricade -> Parricade 1 VOT 0.8 0.6 Fixations to Target 0.4 0.2 0 300 600 900 Time (ms) • Faster activation of target as VOTs near lexical endpoint. • --Even within the non-word range.

46. Eye Movement Results 0 Parakeet -> Barakeet 5 10 15 20 25 30 35 300 600 900 1200 Time (ms) Barricade -> Parricade 1 VOT 0.8 0.6 Fixations to Target 0.4 0.2 0 300 600 900 Time (ms) • Faster activation of target as VOTs near lexical endpoint. • --Even within the non-word range.

47. Experiment 2: Garden Path Analyses b/parricade • Is the latency to switch to the target related to VOT? • Accelerated latency = more (residual) target activation Sparse data but: B: p=.002 P: p=.007

48. Experiment 2 Extensions X X X 2) Effect holds with non-displayed competitors. • Replication: longer • continua (0-45 ms).

49. Experiment 2 Conclusions Gradient effect of within-category variation without minimal-pairs. • Gradient effect long-lasting: mean POD = 240 ms. • Regressive ambiguity resolution: • Subphonemic gradations maintained until more information arrives. • Subphonemic gradation can improve (or hinder) recovery from garden path. (to developmental work)

50. Progressive Expectation Formation • Can within-category detail be used to predict future acoustic/phonetic events? • Yes: Phonological regularities create systematic within-category variation. • Predicts future events.