Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012.

1. Martin J. Pickering, Simon Garrod, Behavioral and Brain Sciences, 2012. An Integrated Theory of Language Production and Comprehension Computer Science & Engineering 2012-20835 Sang-Woo Lee

2. Background - Aphasia

3. Broca’s Aphasia • Expressive aphasia • Agrammatic aphasia • Understand what other people say, but cannot speak the sentence well. • Caused by damage to, or developmental issues in the anterior regions of the brain • Including (but not limited to) the Broca’s area

4. Wernicke’s Aphasia • Also known as Receptive Aphasia • Fluent apahsia, or sensory aphasia • Speak the sentence fluently, but not well-organized sense in their speech • Traditionally associated with neurological damage to Wernicke’s area in the brain • (Actually it is not just simply associated to Wernicke’s area in current experimental result, but anyway…)

5. There are modules which specify some function perception action The “classical Lichtheim-Broca-Wernicke” Model

6. Contents • Background - Aphasia • Traditional independence of production and comprehension • Interweaving in action and action perception • Perception process in action • Predict next action of other • Joint Action • Interweaving in Production and Comprehension • Comprehension process in Production • Predict next speech of other • Interactive Language • Professor’s Question

7. Traditional independence of production and comprehension

8. Traditional model of communication • Discrete stages • A produces, B comprehends • B produces, A comprehends

9. Horizontal Split • Assumes “horizontal split” between production and comprehension • Arrows-within-arrows indicate feedback (in interactive accounts) • But this feedback is internal to production or comprehension • It may involve “general knowledge” • But production does not involve comprehension processes • And comprehension does not involve production processes

10. Example of predict in Behavioral Instance • Interlocutors are not static, as the traditional model assumes, but are “moving targets” performing a joint activitiy (Garrod & Pickering, 2009)

11. Example of predict in Neuroscience (Pickering & Garrod, 2007) Big(neuter) Painting (neuter) Big (common) Bookcase (common) großenBücherschrank großeGemälde

12. Other Counterexample • Also, many experiments demonstrate effects of one on the other • Picture-word interference (Schriefers et al., 1990) • Word identification affected by externally controlled cheek movement (Ito et al., 2009) • And strongly overlapping neural circuits for production and comprehension (e.g., Pulvermüller & Fadiga, 2010; Scott et al., 2009)

13. Result • Prediction process could be naturally understood with production module • When they comprehend the utterance, they also use production model internally. - There is forward model To predict perception caused by their own utterance - Fast alert when you say something wrong

14. Interweaving in action and action perception • Perception process in action • Predict next action of other • Joint Action

15. Interweaving in action and action perception • Close links between action and action perception, e.g. • participants’ arm movements affected by observing another person’s arm movements (Kilner et al., 2003) • And making hand movements can facilitate concurrent visual discrimination of deviant hand postures (Miall et al., 2006) • Such links could have various purposes • Supporting overt imitation • facilitating memory or understanding (“postdictively”) • But authors propose that they aid prediction of own and others’ actions, by use of a forward model • Based on computational neuroscience (Wolpert, 1997; see Grush, 2004)

16. Forward modelling in action • In our terms, the action command causes the action implementer to move the hand and the perceptual implementer to construct the percept • And the efference copy causes the forward action model to generate the predicted hand movement and the forward perceptual model to construct the predicted percept

17. Forward Modeling in Action • Just act • Feel involved percepts of your own act e.g. Own coordination info Feeling of wind blowing to your arm Gravity info …

18. Efference Copy

19. Forward Modeling in Action - Predict perception caused by their own action e.g. Own coordination info Feeling of wind blowing to your arm Gravity info …

20. Prediction-by-Simulation - Predict perception of other’s next action by seeing other’s current action

21. Joint Action • People are highly adept at joint activities (Sebanz et al., 2006). • ballroom dancing, playing a duet, carrying a large object together • Precise timing is crucial • To succeed, A predicts B’s action and B predicts B’s action

22. Joint Action

23. Interweaving in Production and Comprehension • Comprehension process in Production • Predict next speech of other • Interactive Language

24. Forward modeling in language production • Action implementer production implementer • Perceptual implementer comprehension implementer • Action command production command • Drives the production implementer • Efference copy drives the forward models • Comparator monitor • compares the utterance percept and the predicted utterance percept

25. Unifying production and comprehension • Production and comprehension are interwoven • Tight coupling in dialogue (Clark, 1996; Pickering & Garrod, 2004) • Behavioural experiments show effects of comprehension processes on production and vice versa (e.g., Schriefers et al., 1990) • Overlap of brain circuits for production and comprehension (e.g., Pulvermuller & Fadiga, 2010) • Such interweaving facilitates prediction of self and other’s utterances

26. Classical modeling in language production • Just say utterance, • Listen what you say.

27. Forward modeling in language production - Predict perception caused by their own utterance - Fast alert when you say something wrong

28. Self-monitoring • Speaker wishes to say kite • In the past, she has always constructed the kite-concept and then uttered /k/ • She therefore constructs forward model p^[phon](t) = /k/ • If she then incorrectly constructs p[phon] = /g/, the monitor notices the mismatch • If she believes the forward model, she will detect an error (and perhaps reformulate) • Otherwise, she will alter her forward model

29. Prediction-by-simulation - Predict perception of other’s next utterance by listeningother’s current utterance Big(neuter) Painting (neuter)

30. Interactive Language • Joint action involves combining accounts of action and action perception • Similarly, interactive language involves combining accounts of production and comprehension • Facilitates coordination (e.g., short intervals between speakers; Wilson & Wilson, 2005) • Facilitates alignment (developing same representations; Pickering & Garrod, 2004) • Alignment in turn facilitates comprehension (better prediction of others)

31. Interactive Language

32. Conclusion • We propose that language production and comprehension are interwoven • It assumes a central role to prediction in production, comprehension, and dialogue • Speakers construct forward models to predict aspects of their upcoming utterances • Listeners covertly imitate speakers and use forward models to predict the speakers • Our account helps explain the efficiency of production and comprehension and the remarkable fluidity of dialogue

33. Thank you

34. Professor’s Question • Comprehension process in Production • Predict next speech of other • Interactive Language

35. Question 1 • Q1: Give the evidence for how language production and comprehension are tightly interwoven. How does this relate to the perception-action cycle theory of cognitive systems?

36. Question 2 • Q2: Explain and give the evidence for how action, action perception, and joint action are interwoven. Explain how the authors use this to develop accounts of production, comprehension, and interactive language. • Action implementer production implementer • Perceptual implementer comprehension implementer • Action command production command • Comparator monitor • participants’ arm movements affected by observing another person’s arm movements (Kilner et al., 2003) • And making hand movements can facilitate concurrent visual discrimination of deviant hand postures (Miall et al., 2006)

37. Question 3 • Q3: Give examples of what behavioral and neuroscientific data on language processing can be explained by the integrated theory of language production and comprehension explains, while modular theory does not. - Neuroscientific data (Pickering & Garrod, 2007) - Behavioral data