Mental Simulations of Voice and Speech during Reading of Quotations

1. Mental Simulations of Voice and Speech during Reading of Quotations Christoph Scheepers Bo Yao Pascal Belin

2. Introduction • Direct versus indirect speech • Two quoting styles that exist in many languages • Often imply differences in perspective (cf. change in corresponding person and tense features), e.g. • Mary said: “I am hungry”. • Mary said (that) she was hungry. • Important differences in usage • While the main function of indirect speech appears to be to provide a description of what was said by another person, the use of direct speech typically also involves a demonstration (or depiction) of how it was said, e.g. • ..and she was like: “Oh my god! I’m so hungry!” • Direct speech as vivid demonstration • Tannen (1986; 1989) • Clark & Gerrig (1990) • Wade & Clark (1993) • Speakers are more likely to use direct rather than indirect speech quotations when instructed to entertain, as opposed to just inform, a listener

3. Comprehension of Quotations • Implications for reading of quotations? • Intuitive experience of an “inner voice”, especially during silent reading of direct speech quotations • No direct, objective confirmation of this intuition so far • Embodied cognition • Language processing is grounded in mental re-enactment (perceptual simulation) of perceptual states and actions (Barsalou, 1999; 2008) • Given that the use of direct speech is grounded in vivid demonstrations, direct speech should trigger more enriched perceptual simulations (spontaneous auditory imagery) of, e.g., voice and/or manner of speaking than indirect speech • Present investigations • Study 1 (fMRI+eye-tracking): Voice-selective auditory cortex areas (cf. Belin et al., 2000) become more active during silent reading of direct rather than indirect speech quotations • Study 2 (eye-tracking): The context preceding a quotation (implying a “fast-” or a “slow-speaking” quoted speaker) has an influence on how fast one would read a direct-speech quotation, but not how fast one would read an indirect speech quotation

4. Study 1 • Research Question • During silent reading, do the two quotation styles recruit brain areas that are known to be sensitive to human voices differently? • Belin el al. (2000. Nature): Voice-selective areas in human auditory cortex • Focus on top-down activation of voice-selective areas (“inner voice”) during silent reading of direct versus indirect speech quotations • No auditory stimulation! • Hypothesis • Silent reading of direct speech quotations should trigger more top-down activation in those voice-selective areas than silent reading of indirect speech quotations • Because direct quotations are more likely to be grounded in vivid perceptual re-enactments of actual speech (including aspects of voice and manner of speaking) Yao, Belin, Scheepers (2011) Journal of Cognitive Neuroscience

5. Study 1 • Method • Event-related fMRI combined with eye-tracking (silent reading of text) • Participants • 16 native English speakers aged 18 - 44 years, 10 female • Stimuli • 90 short stories containing direct or meaning-equivalent indirect speech quotations (counterbalanced via Latin square) from fictitious protagonists, e.g. • PhD student Ella was summoned to her supervisor Jim’s office to give a report on her current progress. Ella asked for an extension but Jim looked concerned. He said: “Hmm, we really need those data in by next month for that conference.”(direct speech) • PhD student Ella was summoned to her supervisor Jim’s office to give a report on her current progress. Ella asked for an extension but Jim looked concerned. He said that they really needed those data in by next month for that conference. (indirect speech) • 20 vocal (e.g. spoken vowels; laughing; coughing..) and 20 non-vocal (e.g. telephone ringing; dog barking..) sound-clips for a functional voice localizer session after the main reading session Yao, Belin, Scheepers (2011) Journal of Cognitive Neuroscience

6. (1) Main reading session Read stories silently for comprehension and answer yes/no-questions (2) Voice localizer session Listen to 20 vocal and 20 non-vocal sound-clips (8 sec each) Eyes closed! Same as in Belin et al. (2000, Nature) Study 1: Tasks Yao, Belin, Scheepers (2011) Journal of Cognitive Neuroscience

7. Study 1: Results • Auditory voice-localizer regions of interest (ROIs) in green • Within those regions, significantly higher activation (BOLD response) for direct speech than for indirect speech quotations • Measured from 1st fixation into the critical quotation passage to first progressive saccade out of the critical quotation passage (eye-tracking) • Strongly right-lateralized pattern • Two main clusters of enhanced activity for direct (as opposed to indirect) speech quotations were located in voice-selective areas along posterior and middle parts of the right STS • Interestingly, similar right-lateralized activation patterns have previously been found in association with emotional prosody (e.g. Wiethoff et al., 2008; Wildgruber et al., 2006) • No differences in reading rate between direct vs. indirect speech quotations • Mean reading time per word: 204 ms vs. 203 ms • Suggests no difference in processing difficulty between the two quotation styles Yao, Belin, Scheepers (2011) Journal of Cognitive Neuroscience

8. Study 1: Discussion • Evidence for enhanced auditory mental simulation/imagery of voice during silent reading of direct speech quotations (compared to meaning-equivalent indirect speech quotations) • Auditory cortex regions that are particularly ‘interested’ in human voices (cf. Belin et al., 2000) show more top-down activation during silent reading of direct speech quotations • Can auditory voice simulation effects be demonstrated behaviourally as well?...

9. Study 2 • Stimuli • 24 short stories containing direct or meaning-equivalent indirect speech quotations • The context preceding those quotations implied either a fast or a slow-speaking quoted protagonist: • Fast context: Teenage pianist Bobby was to take tart in a local talent competition. He was extremely nervous before his performance. His mother encouraged him… but he was all shaking and said: “No! I can’t do it! This is the end of the journey because I won’t make it this time.” His mother … (direct speech) but he was all shaking and said that he couldn’t do it and that this was the end of the journey because he wouldn’t make it this time. His mother … (indirect speech) • Slow context: At the Royal Infirmary, an old man was dying, and too weak to sit up. He wanted to say something, so his daughter placed a cushion under his head. Slowly, he looked around… and said: “I’m grateful you are all here. This is the end of the journey because I won’t make it this time.” Then he closed his eyes … (direct speech) and said that he was grateful for their coming and that this was the end of the journey because he wouldn’t make it this time. Then he closed his eyes … (indirect speech) Yao & Scheepers (2011) Cognition

10. Study 2 • Method • Oral reading (20 participants): • Read the stories aloud in a random order • Digital audio recordings throughout the session • Reading rate for critical quotation passages analysed in numbers of syllables per second • Silent reading (48 participants): • Read the stories (plus fillers) silently in a random order • Answer yes/no comprehension question after each story • Eye-tracking (EyeLink 1000) throughout the session • Reading rate for the critical quotation passages analysed in terms of ~ first-pass reading time (time from fixating the passage for the first time until another region is fixated) Yao & Scheepers (2011) Cognition

11. Study 2: Hypotheses • Prior research (Alexander & Nygaard, 2008) has demonstrated that oral and silent reading rates are modulated by auditory imagery • People read faster when instructed to imagine familiar ‘fast speakers’ as authors of text for reading • Here, we do not use instructions, but rather look at spontaneous reading rate adjustments as a function of quoting style (direct / indirect speech) and context (implying fast or slow quoted speakers) • Inspired by Alexander & Nygaard (2008), we expect reading rates to be modulated by context (fast or slow quoted speaker) • Informed by Study 1, we expect contextual modulation of reading rate to be stronger for direct than for indirect speech quotations Yao & Scheepers (2011) Cognition

12. Study 2: Results • Reading aloud task: Oral reading rates (in syllables/second) for the critical quotation passages • Clear Context × Quoting Style interaction • Context-fast > Context-slow in direct speech • No such context effect in indirect speech • With direct speech quotations, participants spontaneously “act out” the contextually implied speech rate of the quoted speaker • Silent reading task (eye-tracking): First-pass reading times (in ms) for the critical quotation passages • Clear Context × Quoting Style interaction • Context-fast < Context-slow in direct speech • No such context effect in indirect speech • With direct speech quotations, silent readers adjust their reading rates to the contextually implied speech rate of the quoted speaker Yao & Scheepers (2011) Cognition

13. Conclusions • Converging evidence from fMRI and eye-tracking, suggesting that readers are more likely to mentally simulate (or spontaneously imagine) aspects of the quoted speaker’s voice (and manner of speaking) during silent reading of direct speech quotations • Consistent with “direct speech as vivid demonstration” (Clark & Gerrig, 1990) • Perceptual grounding (cf. Barsalou, 1999; 2008) of direct speech: • Since direct speech typically occurs in the context of vivid demonstrations/re-enactments of actual speech acts, a direct quotation style is more likely to be taken as a cue to engage in enriched perceptual simulations of voice than an indirect quotation style (focus on content) • … and maybe not only of voice! (to be continued…)