Brain signatures of game play and language processing

1. Brain signatures of game play and language processing • Robert V. Reichle • rreichle@niu.edu • Department of Foreign Languages • Center for the Interdisciplinary Study of Language and Literacy • Northern Illinois University

2. L2 PROCESSING • For late L2 learners, L2 processing is difficult • Corresponding features or constructions in the native language do not ensure success in the L2 • Variability is typical • Native-like attainment is not typical • Previously thought to relate to putative critical period for L2 acquisition

3. L2 PROCESSING • Among other explanations for divergence in L1/L2 processing, Ullman’s (2001, 2004) Declarative/Procedural model • Two domain-general memory systems • Declarative: facts, associations between form and meaning, non-compositional words • Procedural: rule-governed processes, morphosyntactic composition, cognitive-motor skills • Some L2 learners become more nativelike as they come to rely more on procedural memory in the L2

4. EVENT-RELATED POTENTIALS (ERPs) Lexical/semantic processing: N400 Morphosyntactic/grammatical processing: LAN • (e.g., graphs from Osterhout et al., 2006, p. 204 and Neville et al., 1991; Friederici, Steinhauer, & Frisch, 1999; Friederici, 2002; Hahne & Friederici, 1999, 2001; Hagoort & Brown, 1999; Kaan et al., 2000; Kutas & Hillyard, 1980; Osterhout, Bersick, & McLaughlin, 1997; Osterhout & Holcomb, 1992; Osterhout & Mobley, 1995; Steinhauer, Halter, & Friederici, 1999; Friederici, Pfeifer, & Hahne, 1993; Kluender & Kutas, 1993; Münte, Heinze, Matzke, Wieringa, & Johannes, 1998) • Differential ERPs for L1 syntax/semantics interpreted as evidence for DP model (Ullman, 2004)

5. EVENT-RELATED POTENTIALS (ERPs) • Proficiency effects on sensitivity to inflectional morphology and syntax • Artificial language learning: with language training, N400 LAN/P600 • (e.g., Friederici, Steinhauer, & Pfeifer, 2002; Morgan-Short, Sanz, Steinhauer, & Ullman, 2010; Morgan-Short, Steinhauer, Sanz, & Ullman, 2012; Morgan-Short, Finger, Grey, & Ullman, 2012) • Real language learning: with increasing proficiency, N400 LAN/P600 • (e.g., Bowden et al., 2007; Hahne et al., 2006; Osterhout et al., 2006; Rossi et al., 2006; Steinhauer et al., 2006; Gillon Dowens, Vergara, Barber, & Carreiras, 2010; McLaughlin, Osterhout, & Kim, 2004; McLaughlin, Tanner, Pitkänen, Frenck-Mestre, Inoue, Valentine, & Osterhout, 2010; Tanner, Osterhout, & Herschensohn, 2009; Ojima, Nakata, & Kakigi, 2005)

6. EVENT-RELATED POTENTIALS (ERPs) • Proficiency effects on sensitivity to inflectional morphology and syntax • Artificial language learning: with language training, N400 LAN/P600 • (e.g., Friederici, Steinhauer, & Pfeifer, 2002; Morgan-Short, Sanz, Steinhauer, & Ullman, 2010; Morgan-Short, Steinhauer, Sanz, & Ullman, 2012; Morgan-Short, Finger, Grey, & Ullman, 2012) • Real language learning: with increasing proficiency, N400 LAN/P600 • (e.g., Bowden et al., 2007; Hahne et al., 2006; Osterhout et al., 2006; Rossi et al., 2006; Steinhauer et al., 2006; Gillon Dowens, Vergara, Barber, & Carreiras, 2010; McLaughlin, Osterhout, & Kim, 2004; McLaughlin, Tanner, Pitkänen, Frenck-Mestre, Inoue, Valentine, & Osterhout, 2010; Tanner, Osterhout, & Herschensohn, 2009; Ojima, Nakata, & Kakigi, 2005)

7. brocanto training game • Artificial language (BROCANTO) • Fulfills syntactic requirements of natural language • Contains morphosyntactic features that differ from L1 of participants • Participants trained in lexicon of artificial language • One group trained to proficiency in BROCANTO via a two-player computer board game • Played against each other in pairs • Orally expressed all game moves in BROCANTO • Received feedback on valid game moves and grammatical BROCANTO utterances • The structure of the language not dependent on the structure of the game • (Friederici, Steinhauer, & Pfeifer, 2002; for similar BROCANTO2 design, see also Morgan-Short, Sanz, Steinhauer, & Ullman, 2010; Morgan-Short, Steinhauer, Sanz, & Ullman, 2012; Morgan-Short, Finger, Grey, & Ullman, 2012)

8. brocanto training game aaf trul prez nöri aak füne plox the trul-piece captures horizontally the round plox-piece aaf trul prez nöri *rix füne ploxthe trul-piece captures horizontally *buy round plox-piece • ERPs • ERP data collection using syntactically felicitous and anomalous items took place after participants in the training group reached a 95% accuracy threshold. • Training group had biphasic response to violations that resembled LAN/P600 of L1 processing

9. brocanto training game • Why was this a successful training paradigm?

10. brocanto training game • Why was this a successful training paradigm? • Gee (2003): active, critical learning • cooperation • competition • a requirement of successful learning in order to progress • feedback • meta-awareness of gameplay mechanics

11. brocanto training game • Why was this a successful training paradigm? • Declarative/Procedural model • nativelike syntactic processing should be handled by procedural memory • learning a cognitive-motor skill (like the rules of the training game) should also be handled by this system • Did these two types of learning engage similar cognitive processes? Does “words and rules” have an equivalent “sprites and rules” (Reichle, 2012)? • P600 not necessarily fully language-specific — also observed for structural revision in music (e.g. Patel et al., 1998)

12. Research questions • Does the execution of gameplay skills engage the same processes as syntactic processing? • Specifically, examining violations of syntax and violations of expected gameplay behavior. • If the same processes, then could aid learning, but could also present confound to interpretation of results. • Does processing linguistic stimuli in the context of a game lead to a task effect compared to linguistic stimuli in isolation?

13. Hypotheses • Violations of expected gameplay behavior will elicit LAN and P600 effects comparable to those seen for violations of morphosyntax. • Processing violations of morphosyntax in the context of a game causes a task effect compared to violations of morphosyntax in isolation.

14. ERP Experiment 1 • 23 participants (5 rejected due to artifacts) • native speakers of English • pre-screened for knowledge of rules of movement in chess • EEG data were recorded from nine sites (F3, Fz, F4, C3, Cz, C4, P3, Pz, P4)Four conditions • A: Control. Legal moves followed by grammatical sentences • B: Morphosyntax violation. Legal moves followed by sentences with a morphosyntax violation • C: Gameplay mismatch. A chess move that is legal for a piece other than the one shown in the image, followed by a grammatical sentence • D: Gameplay violation. A chess move that is never legal for any piece, followed by a grammatical sentence

17. A: The white queen takes the black pawn.

18. B: The white queen *taking the black pawn.

21. C: The white queen takes the black pawn.

24. D: The white queen takes the black pawn.

25. ERP Experiment 1 • Stimulus presentation • each image onscreen for 1 sec • 600ms pause • word-by-word presentation: 290 ms plus 30 ms per letter, up to a maximum of 590 ms, with a 150 ms interval between words • Analysis • At fourth word (takes/*taking) • Mean amplitude data in 200-500ms and 500-800ms windows • Repeated-measures ANOVAs: 4 (Condition) x 3 (Anteriority) x 3 (Laterality)

26. F3 Fz F4 C3 Cz C4 P3 Pz P4

28. F3 Fz F4 Omnibus 200-500ms: Cond * Lat, p < .05 C3 Cz C4 P3 Pz P4

29. Morph. vs Control 500-800ms: Cond * Lat, p < .05 F3 Fz F4 C3 Cz C4 P3 Pz P4

30. Game rule mismatch vs Control 200-500ms: NS 500-800ms: NS F3 Fz F4 C3 Cz C4 P3 Pz P4

31. Game rule violation vs Control 500-800ms: Cond * Ant * Hem, p = .022 F3 Fz F4 C3 Cz C4 P3 Pz P4

32. Results • P600 effect for morphosyntax violations • No effect for game rule mismatch • Game rule violation • Negativity effect, strongest in left and mid central and frontal sites • Interpreted as possible variation of LAN • Extended into later time window than typical biphasic LAN • But, not biphasic — no P600 effect followed

33. Discussion • Hypothesis: Violations of expected gameplay behavior will elicit LAN and P600 effects comparable to those seen for violations of morphosyntax. • Not fully supported — no P600. Consistent with generally language-specific nature of P600. • Partially supported by presence of LAN. But work still remains on reproducibility and functional significance of LANs (e.g. Steinhauer, Drury).

34. Discussion • Implications • Possibility remains that processing gameplay rules engages similar cognitive processes as processing morphosyntax (LAN) • No evidence that gameplay elicits P600 • Doubtful that P600s seen in BROCANTO studies were indices of gameplay-related processing

35. Hypotheses • Violations of expected gameplay behavior will elicit LAN and P600 effects comparable to those seen for violations of morphosyntax. • Processing violations of morphosyntax in the context of a game causes a task effect compared to violations of morphosyntax in isolation.

36. ERP Experiment 2 • Show morphosyntactic anomalies within and without game context • 2 groups • Image group: Saw chess move after reading sentence • NoImage group: Did not see chess moves • 12 participants (6 per group) • native speakers of English • pre-screened for knowledge of rules of movement in chess • EEG data were recorded from nine sites (F3, Fz, F4, C3, Cz, C4, P3, Pz, P4)

37. ERP Experiment 2 • 2 Conditions • Control: The white queen takes the black pawn. • Morphosyntax violation: The white queen taking the black pawn. • 2 fillers • Progressive filler: The queen is takingthe black pawn. • Game rule violation: The black queen takes the white pawn.

38. ERP Experiment 2 • Stimulus presentation • word-by-word presentation: 290 ms plus 30 ms per letter, up to a maximum of 590 ms, with a 150 ms interval between words • for Image group, presentation of game images 1 sec after final word • Analysis • At fourth word (takes/taking) • Mean amplitude data in 200-500ms and 500-800ms windows • Repeated-measures ANOVAs: 2 (Condition) x 3 (Anteriority) x 3 (Laterality)

39. Control Morphosyntax violation NoImage group

40. Control Morphosyntax violation Image group

41. Control Morphosyntax violation Between-group omnibus: 400-900ms: Cond * Group, p = .26 Image group

42. Control Morphosyntax violation Image group: 400-900ms: Cond, p = .25 400-900ms: Cond * Lat, p = .19 400-900ms: Cond * Lat * SRTT, p = .2 Image group

43. Results • Trend toward between-group difference (task effect) • Trend toward P600 for Image group • Data collection still in progress — more participants needed

44. Discussion • Hypothesis: Processing violations of morphosyntax in the context of a game causes a task effect compared to violations of morphosyntax in isolation. • Preliminary results trend toward support of hypothesis. • Attributable to “motivation” in the broadest sense. Additional studies needed to separate out Gee’s factors (competition, meta-awareness, etc.) from simpler explanations (i.e. more attentive due to presence of images)

45. CONCLUSIONS • Evidence of an LAN for the processing of gameplay violations suggests gameplay may engage cognitive processes also used in language processing. • Lack of P600 for gameplay processing indicates that L2 studies using game training paradigms are not likely to confound indices of gameplay with indices of morphosyntactic processing. • Preliminary data suggest trend toward game-related task effect, suggesting increased motivation or attention within game context. • Additional studies needed to further tease apart variables.

46. Acknowledgments • This research has been supported by the NIU Division of Research and Graduate Studies, the NIU Center for the Interdisciplinary Study of Language and Literacy, and the NIU Center for the Study of Family Violence and Sexual Assault. • The author thanks Cody Happ and Regina Hiraoka for their assistance with materials creation and data collection, and all participants for their cooperation.

47. Thank you