Disfluency isn’t just um and uh: the role of prosody in the comprehension of disfluency

1. Disfluency isn’t just um and uh: the role of prosody in the comprehension of disfluency Jennifer E. Arnold, Becky Altmann, & Michael K. Tanenhaus University of Rochester

2. What are the cognitive processes of reference comprehension? …the apple… …thee uh apple…

3. Accented NPs - Given unfocused accessible • Put the grapes above the candle. • Now put the CA…. (Dahan et al. 2002) What makes things accessible for reference comprehension? What makes things accessible for reference comprehension?

4. Disfluency - New accessible • Put the grapes above the candle. • Now put theee uh CA…. (Arnold, Fagnano, and Tanenhaus) What makes things accessible for reference comprehension? What makes things accessible for reference comprehension?

5. Disfluency more likely for NPs with new referents (data from Arnold, Wasow, Ginstrom, and Losongco, 2000)

6. Discussion Questions • If disfluencies are helpful in distinguishing new target objects, why are they only seen in 6% of utterances? • Why do some people show far fewer disfluencies in their speech than others? Can people be trained to be more fluent?

7. Manifestations of Disfluency • Um, uh • Um - reflects a big problem • Uh - reflects a small problem (Clark & Fox Tree) • Theee • Pauses • Correlated prosodic changes

8. Comprehenders use disfluency on-lineArnold, Fagnano & Tanenhaus, CUNY 2002 Put the grapes below the camel.Now put theee, uh ….

9. Arnold, Fagnano & Tanenhaus, CUNY 2002 FLUENT / GIVEN Put the grapes above the camel. Now put the CAMEL . . . FLUENT / NEW Put the grapes above the candle. Now put the CAMEL . . . DISFLUENT / GIVEN Put the grapes above the camel. Now put thee uh CAMEL . . . DISFLUENT / NEW Put the grapes above the candle. Now put thee uh CAMEL . . .

10. Arnold, Fagnano & Tanenhaus, CUNY 2002 FLUENT / GIVEN Put the grapes above the camel. Now put the CAMEL . . . FLUENT / NEW Put the grapes above the candle. Now put the CAMEL . . . DISFLUENT / GIVEN Put the grapes above the camel. Now put thee uh CAMEL . . . DISFLUENT / NEW Put the grapes above the candle. Now put thee uh CAMEL . . .

11. Arnold, Fagnano & Tanenhaus, CUNY 2002 FLUENT / GIVEN Put the grapes above the camel. Now put the CAMEL . . . FLUENT / NEW Put the grapes above the candle. Now put the CAMEL . . . DISFLUENT / GIVEN Put the grapes above the camel. Now put thee uh CAMEL . . . DISFLUENT / NEW Put the grapes above the candle. Now put thee uh CAMEL . . .

12. Arnold, Fagnano & Tanenhaus, CUNY 2002 FLUENT / GIVEN Put the grapes above the camel. Now put the CAMEL . . . FLUENT / NEW Put the grapes above the candle. Now put the CAMEL . . . DISFLUENT / GIVEN Put the grapes above the camel. Now put thee uh CAMEL . . . DISFLUENT / NEW Put the grapes above the candle. Now put thee uh CAMEL . . .

13. Comprehension Study Methods • Instructions recorded by experimenter • Subjects told they were recorded by another subject • Disfluencies sounded natural • Lexical retrieval was a plausible source of disfluency

15. Method Questions • What problems do you think it may present having the lead author perform the verbal stimuli the subjects heard? • In this experiment the instructions were prerecorded and performed by the experimenter. Is it possible that "performing" a disfluency and producing a disfluency in real time speech are different? Were poor takes excluded, and if so, on what grounds? • Was there any control for students who had taken BCS courses with you or possibly had read the Dahan et. al. paper and would then know what they were participating in?

16. Results: • Baseline differences “zeroed out” • Examine looks to target and competitor, considering cases where the subject was not already looking at either cohort • 64% of data in analysis

17. Exp. 1 Results: “Target Advantage” 64% of data

18. GIVEN NEW FLUENT/ NEW DISFLUENT/ GIVEN … above the candle. … above the camel. Now put the CAMEL . . . Now put theee uh CAMEL . . . New cohort Given cohort Unrelated

19. FLUENT all new items Now put the CAMEL . . . all given items all new items DISFLUENT all given items Now put thee, uh CAMEL . . . Baseline differences: Disfluency causes New bias

20. Disfluency results in multiple changes to speech signal LEXICAL-SEGMENTAL: thee uh the “THINKING PROSODY”: longer shorter excursion flat disfluent fluent

21. Results summary • Fluent instructions • given bias at head noun • Disfluent instructions • new bias at head noun • Early new bias after Now • Potential prosodic signal of disfluency

22. Pitch affected degree of new bias All given objects All new objects

23. Exp. 2: What is the role of pitch? • Pitch excursioncrossed withdisfluency • Large vs. small • Disfluent vs. fluent • Given vs. new

24. One “Now put” for each condition N o w p u t Now put Fluent/ large excursion Disfluent/ large excursion N o w p u t Now put Fluent/ small excursion Disfluent/ small excursion

25. Large excursion supports new biasSmall excursion supports given bias 63% of data

26. Conditions with mismatching prosody are intermediate

27. NEW NEW DISFLUENT / LARGE PITCH EXCURSION NEW GIVEN … above the candle. … above the camel. Now put theee uh CAMEL . . . Now put theee uh CAMEL . . . competitor looks unrelated looks target looks

28. GIVEN GIVEN DISFLUENT / SMALL PITCH EXCURSION NEW GIVEN … above the candle. … above the camel. Now put theee uh CAMEL . . . Now put theee uh CAMEL . . . competitor looks unrelated looks target looks

29. New bias at head nouncomes from both pitch and lexical-segmental manifestations of disfluency • FLUENT:Small excursion supports GIVEN BIAS • DISFLUENT:Large excursion supports NEW BIAS

30. Brief initial New bias for Large excursion Subanalysis: Looks to new objects when subject was not fixating a new object at onset of Now 69% of data

31. New bias in Disfluent conditions FLUENT all new items Now put the CAMEL . . . all given items all new items DISFLUENT all given items Now put thee, uh CAMEL . . .

32. Preliminary Offline Results:What do you think the speaker will refer to next?“Put the grapes above the camel. Now put…“Put the grapes above the camel. Now put the/thee uh…

33. Results summary • Disfluency + pitch excursion = new bias • Disfluent “thee uh” alone = no new bias • Fluent pitch excursion alone = no new bias

34. “Thinking Prosody” • Contributes to impression of speaker difficulty • Production difficulty causes multiple changes to the speech signal • Disfluent words (um, uh) • Repeats, repairs • Choices in consituent ordering (Arnold et al., 2000) • Choices about where to put intonation breaks (Watson 2002) • Pitch excursion / lengthening

35. Conclusions • Multiple cues to speaker difficulty lead to a new bias • Disfluency as a nonlocal cue

36. Thanks to the following people for advice, discussion, and / or help with data collection • Katherine Crosswhite • Joyce McDonough • Maria Fagnano • Dana Subik • Duane Watson

37. EXTRA SLIDES

38. How did these materials compare with those for Experiment 1?

41. GIVEN GIVEN FLUENT / SMALL PITCH EXCURSION NEW GIVEN … above the candle. … above the camel. Now put the CAMEL . . . Now put the CAMEL . . . competitor looks unrelated looks target looks

42. GIVEN GIVEN FLUENT / LARGE PITCH EXCURSION NEW GIVEN … above the candle. … above the camel. Now put the CAMEL . . . Now put the CAMEL . . . competitor looks unrelated looks target looks

43. New bias more evident in Disfluent / Large All given objects All new objects

44. Experiment 1: Results at Head Noun competitor looks unrelated looks target looks