Introduction Two known ERP mechanisms for language comprehension include:

1. N400 P600 -3uv -3uv 400 400 800 800 1200 1200 Semantic integration in real-world and cartoon-world movie clips: Electrophysiological evidenceCourtney Brown1, Tatiana Sitnikova2, Gina R. Kuperberg1,2, Phillip J. Holcomb11 Department of Psychology, Tufts University, Medford MA2. Department of Psychiatry and the Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA Experiment 1: Functionally and Semantically Incongruent Events • Introduction • Two known ERP mechanisms for language comprehension include: • N400: processing semantic or concept-based violations • Late positive component (LPC): responding to syntactic or rule-based violations (Kuperberg, 2007, Sitnikova et al, 2006) • A more anterior N400 effect, and picture-specific N300 waveform has been illustrated in reference picture processing, including static picture discourse with incongruent final scenes (West & Holcomb, 2002). • Initial work from Sitnikova et. al (2003) shows both an N400 and LPC (not shown in previous picture studies), in response to real-word videos of common activities, time-locked to onset of incongruent object • Current research sought to further examine the origins and the limits of both waveforms in real-world discourse processing • Videos were created with semantically incongruent events (man rolling dough after context of shaving), as well as functionally and semantically incongruent events (man using rolling pin as a shaving tool). This contrast would isolate effects due to semantic incongruity alone. • Cartoons were utilized to test boundaries of real-world comparisons with actions that were semantically congruous, yet functionally impossible (Wyle Coyote runs off a cliff and hovers in mid-air). This variable could then isolate any functional incongruity effects (as opposed to the movies, where functional incongruities retained a semantic incongruity component) • Results • Repeated Measures ANOVA with Greenhouse-Geisser correction: • Experiment 1: Semantically and Functionally Incongruent Events • In the 250- 350, and 350- 600ms epochs, there was an increased negativity that attained significance predominantly within the anterior/frontal regions of the scalp • In the 600- 1000 ms epoch, there was an increased positivity in parietal/occipital regions that attained significance. There was also a significant continuation of the earlier negative effect in anterior regions. • Experiment 2: Semantically Incongruent Events • In the 250- 350 and 350- 600ms epochs, significantly increased negativities were observed in the anterior through central regions of the scalp • In the 600- 1000 ms epoch, the anterior difference in negativity attained significance, and no significant effects were duplicated in the parietal regions • Experiment 3: Semantically Incongruous Events, Physically Impossible Events in Cartoons • In the 250- 350 ms and 350- 600ms epochs, Group by Electrode effects (negativity in semantic condition, anterior areas) were significant for the Semantic condition, while Electrode effects were seen in both conditions • In the 600- 1000 ms, the Semantic condition was significant for Group x Electrode Effects. The Physical condition attained significance for group (Physical condition more positive than Normal condition), Electrode (Anterior and occipital areas more positive), and Group x Electrode effects Fz Pz Experiment 2: Semantically Incongruent Events N300 Fz Fz Pz Pz No Significant P600 Effect

2. Discussion • Significant N400 effect was elicited by: • Semantically incongruent target actions in movies and cartoons • Movie target actions that were semantically and functionally incongruent with preceding context. • This effect was most likely driven by the semantic aspect of the object incongruity (a rolling pin is not part of a bathroom/shaving routine) • Significant effect was NOT elicited by: • Cartoons that were semantically congruous, but functionally impossible by reality-based rules of physics • The anterior-central N400 effect in the 3 studies, as well as an earlier, N300 peak supports previous picture-based research 3, 5 • Significant LPC was elicited by: • Movie condition where there was a functional/object property violation in addition to semantic violation of expected target action • Here, the object property/ functional violations were more likely driving effects (a rolling pin, by nature, does not have a sharp edge to shave) • Cartoon condition involving an object property/functional violation, but no semantic violation regarding expected target action • LPC was not elicited by: • Conditions where there was only a semantic incongruity with the context • The posterior distribution of the LPC correlates with prior real-world event research 3 N400 in Semantic Incongruence Condition P600 in Physical Incongruent Condition -3uv 400 800 1200 Experiment 3: Normal- Physical Incongruities • Methods • Experiment 1: Semantically and Functionally Incongruent Events • 15 Right handed, native English speaker participants (8 female, mean age =18.5 years) • Subjects viewed video clips of common activities (fixing flat tire, shaving) ranging between 5- 24 seconds • Asked to button-press after question mark as to whether preceding sequence would be witnessed in real-life • ERPs to 64 scalp electrodes were recorded to onset of final scene • Experiment 2: Semantically Incongruent Events • 15 Right handed, native English speaker participants (8 female, mean age =18.5 years) • Target scenes from congruent clips switched in order to create semantically incongruous condition • Experiment 3: Semantically Incongruous Events, Physically Impossible Events in Cartoons • 18 Right-handed, native English speaker participants (9 female, mean age = 21 years) • Cartoon clips (5.5- 20.9ms) involved a context scenario, 500 ms blank screen, then an angle change for the target action • Subjects asked to place clips into one of 3 categories: • Normal: Character’s action was semantically expected and physically possible • Semantic: Character remained the same, but target action from different scenario and therefore semantically unexpected • Physical: Action was semantically congruous with context, but physically impossible • ERPs to 32 scalp electrodes were recorded to onset of final scene Experiment 3: Semantically Incongruous Events, Physically Impossible Events in Cartoons Fz Pz • Conclusions • The findings support the idea of distinct roles for these waveforms in real-world processing, similar to those seen in prior language research. • The N400 is a known processor of semantic/concept relationships in language2. The anterior N400 to incongruent target scenarios, regardless of object function discrepancies, supports the idea of N400 as a “first pass” integrating context and target action. It also supports prior research illustrating that visual images activate semantic representations distinct to those activated by words 3,4,5. • The LPC is a processor of syntactic function in sentences, and also in comparing noun phrase properties to central function requirements of verb1. These results support the role of the LPC in “second pass” semantic processing of functional/object properties of real-world events, in terms of object properties matching intended action/properties of a character. • Given a context devoid of real-world restrictions (cartoons), subjects still exhibited the LPC, suggesting that real-world conceptions retain influence in this processing. Experiments 1-3: Voltage Distributions Experiment 1: Congruent- Incongruent Objects Experiment 2: Congruent- Incongruent Scenes Movie Set-up Cartoon Set-up 250- 350 ms 350- 600 ms 600- 1000 ms 250- 350 ms 350- 600 ms 600- 1000 ms Physically Incongruent Semantically Incongruent Semantically and Functionally Incongruent Semantically Incongruent + + + + Late Positivity to Incongruent Objects or Character Properties N400 to Incongruent Target Scenes Cut and Angle Change Cut and Angle Change References 1 Kuperberg G., 2007. Brain Research. 2 Kutas M., Hillyard S.A., 1980. Reading senseless sentences: brain potentials reflect semantic incongruity. Science 207, 203-205. 3 Sitnikova, T., Kuperberg, G., & Holcomb, P. J., 2003. Semantic integration in videos of real- world events: An electrophysiological investigation. 4 Sitnikova T., West W.C., Kuperberg G.R., Holcomb P.J., 2006. The neural organization of semantic memory: Electrophysiological activity suggests feature-based segregation. Biological Psychology 71, 326-340. 5 West, W. C., & Holcomb, P. J., 2002. Event- related potentials during discourse- level semantic integration of complex pictures. Cognitive Brain Research, 13, 363- 375. Experiment 3: Normal- Semantic Incongruities ? ? ? ?