Grounding Knowledge in the Brain’s Modal Systems

Grounding Knowledge in the Brain’s Modal Systems Lawrence W. Barsalou Department of Psychology Emory University June 2010 Research supported by National Science Foundation grants SBR-9421326, SBR-9796200, SBR-9905024, BCS-0212134 DARPA grants FA8650-05-C-7256, FA8650-05-C-7255 Emory fMRI seed grants Grounding Knowledge

The conceptual system 1. represents knowledge about experience in the world 2. organizes knowledge categorically concepts in memory represent categories in the world 3. provides representational support across cognitive tasks online processing high-level perception categorization of perceived entities and events inferences that go beyond the information given offline processing in memory, language, and thought conceptualization of entities not present guides learning provides interpretations of novel material expertise grows with conceptual vocabulary Grounding Knowledge

The traditional approach: Semantic memorye.g., Tulving, 1972; Collins & Loftus, 1975 • modular • distinct from episodic memory and the brain’s modal systems (e.g., vision) • amodal • non-perceptual representations Grounding Knowledge

The transduction principle in amodal conceptual systems amodal symbols are transduced from modal states constitute knowledge about categories a modular system with unique operating principles legs tail barks pat soft Grounding Knowledge

The transduction principle in amodal symbol systems • amodal symbols are not linguistic symbols • the conceptual symbols that underlie language • transduction underlies many common approaches to representation • feature lists, semantic nets, schemata, frames, production systems, etc. § ¥ Þ ŧ Э Grounding Knowledge

§ ¥ Þ ŧ Э “dog” Representing knowledge in amodal symbol systems • amodal symbols later represent categories in their absence • constitute the knowledge that underlies memory, language, thought • no representations in modal systems required or involved Grounding Knowledge

An alternative approach • non-modular • concepts utilize sensory-motor and other modal systems (e.g., affect) • modal • modal simulations represent concepts Grounding Knowledge

Capturing neural activity in the brain’s modal systemsDamasio (1989), Barsalou (1999, 2003, 2005), Simmons & Barsalou (2003) • modal states are captured during online experience • by conjunctive neurons in hierarchically-organized association areas • capture is partial • not complete Grounding Knowledge

“dog” Running simulations to represent knowledge • simulations (reenactments) represent categorical knowledge • may often be unconscious, not necessarily conscious (as in imagery) • always partial, may be distorted • could be exemplars, averages of exemplars, etc. Grounding Knowledge

Empirical evidence for simulation:A general computational mechanism in the brain • evidence across disciplines • cognitive psychology • social psychology • developmental psychology • cognitive neuropsychology • cognitive neuroscience • evidence across processes • perception (perceptual anticipation) • working memory (imagery and rehearsal) • implicit memory (sensory-motor priming) • explicit memory (recollection) • knowledge representation (conceptualization) • language (meaning) • thought (envisioning possible scenarios) • social cognition (mirroring and empathy) • For a recent review see: • Barsalou, L.W. (2008). Grounded cognition. • Annual Review of Psychology, 59, 617-645. • Conjecture: • Multiple control systems. • One representation system. Grounding Knowledge

Common misperceptions • NOT a classic empiricist theory • in principle, simulations for categories could be genetically determined • strong genetic constraints determine feature systems and association areas • anticipating important categories in evolutionary history • NOT a recording system, symbolic operations are central (Pylyshyn, 1973) • doesn’t simply capture records of experience • instead, interpretation of experience lies at the core of this account • via symbolic operations • implemented with mechanisms not presented today • knowledge does NOT solely reflect perception of the external world • also perception of mental states, meta-cognition, affect, etc. (“introspection”) • central to abstract concepts Grounding Knowledge

Overview of research to be presented 1. Examples of simulation 2. Simulation in situated action 3. Simulation in natural abstract categories 4. Simulation in symbolic operations predication conceptual combination Grounding Knowledge Grounding Knowledge 12

Shape inferences in comprehensionZwaan, Stanfield, and Yaxley (2002) hypothesis if readers simulate the meaning of a text to understand it, then text representations should have perceptual properties, even when these properties are not mentioned method a sentence was presented a picture was presented and participants had to name it as quickly as possible key manipulation whether or not the pictured object had a shape that matched the shape of the object implied in the sentence Examples of sentences Examples of pictures The bird sat quietly in the tree (implies a bird with its wings folded) The bird flew quickly across the sky (implies a bird with its wings flapping) Not Mentioned Mentioned Grounding Knowledge

Results conclusion when comprehending sentences, participants simulated the scenes described,thereby committing to a particular shape of the objects mentioned Naming RT (ms) Picture Shape Grounding Knowledge

Simulating actions to represent verb meaningHauk, Johnsrude, and Pulvermüller (2004) method participants read isolated words in an fMRI scanner (2.5 sec rate) subsets of words referred to face, arm, or leg movements (mixed with non-motor words) e.g., “lick,” “pick,” “kick” participants later performed actual motor movements (localizer task) i.e., moved their tongue, finger, or foot prediction if simulations represent word meanings,then words for different body part movementsshould activate the respective regions of the motor system these activations should also lie near thosefor the localizer task Grounding Knowledge

Results the predicted somatotopic order of word activations appeared activations occurred in the motor system for the action words relative to reading strings of hash marks leg activations were vertically highest, then arm, then face • leg and arm activations overlapped for the word and localizer tasks • face discrepancies could indicate less correspondence between words and the localizer task Grounding Knowledge

Conceptual processing is situatedBarsalou (2003), Smith & Semin (2004), Yeh & Barsalou (2006) situations frame conceptual representations concepts are not learned and represented in a vacuum concepts are learned and represented in a situated manner background situations prepare agents for situated action provide useful inferences about: settings agents and objects actions and events mental states tailored to different courses of situated action for the same concept e.g., chairs in living rooms vs. offices vs. jets situational inferences delivered via multimodal simulations across the relevant modalities Grounding Knowledge

Situating physical objectsWu and Barsalou (2009) task and results participants produced the features of objects (e.g., apple) produced non-requested information about settings and mental states suggests that they situated their conceptualizations of the objects Grounding Knowledge

Situating manipulable objectsChao & Martin (2000) participants viewed manipulable objects activation occurred in motor and parietal areas associated with manipulating objects (but did not occur for non-graspable objects) participants situated the manipulable objects with respect to action on categorizing a visual picture, motor inferences were produced review of related findings Lewis (2006) Grounding Knowledge

Taste inferences for foodsSimmons, Martin, & Barsalou (2005) • presented participants with pictures of foods and houses • relatively tasty foods from the undergraduate perspective • no fruits and vegetables • 1-back task • food pictures should activate categorically-related inferences • taste areas should become active Situated Simulation

Activations in primary gustatory cortex for foods (frontal operculum) Z = 20 Z = 16 Z = 10 Z = 9 Z = 5 Z = -9 64, -4, 20 tasting sucrose - deAraujo et al. (2003), p.2063 - R. Operculum 36, 0, 16 tasting chocolate - Small et al. (2001), p.1724 - R Insula/Operculum 54, 12, 10 tasting umami - deAraujo et al. (2003), p.316 - R Insula/Operculum 45, 3, 5 tasting glucose - Francis et al. (1999), p.457 - operculum 45, 1, -9 tasting sucrose - deAraujo et al. (2003), p.2063 – Anterior Insula 36, -6, 9 viewing food pictures - Simmons, Martin, & Barsalou- R Insula/operculum Situated Simulation

Activations in the taste reward area for foods (orbitofrontal cortex) Z = -30 Z = -24 Z = -20 Z = -18 Z = -10 Z = -6 -4, 51, -30 abstract reward - O'Doherty ,et al. (2001) -10 ,42, -24 abstract reward - O'Doherty ,et al. (2001) -4, 30, -20 abstract reward - O'Doherty ,et al. (2001) -9, 26, -18 tasting glucose - Frances, Rolls, et.al. (1999) -32, 50, -10 flavor center - de Araujo et, Rolls, Kingelbach, et al. (2003) -18, 48, -10 property verification, - Simmons, Pecher, Hamann, et al. (submitted) -34, 26, -6 tasting umami- de Aaujo, et al. (2003) -21, 33, -18 viewing food pictures- Simmons, Martin, & Barsalou Situated Simulation

Situating social objectsGil-da-Costa, Braun, Lopes, Hauser, Carson, Herscovitch, & Martin (2004) monkeys listened to recorded coos and screams of other monkeys compared PET activations to those for unfamiliar sounds (musical instruments) results activations in auditory areas (perception) activations for situated inferences visual areas inferior temporal (faces) superior temporal (expression, motion) frontal and limbic areas medial prefrontal (mental states?) amgydala (emotion) hippocampus (emotional memory) the monkeys simulated the situations associated with the sounds provides continuity with the human conceptual system (Barsalou, 2005) Grounding Knowledge

Representing abstract concepts with simulationBarsalou (1999) • concepts are typically situated • not represented in vacuum, but in a setting with agents, objects, events, etc. • concrete concepts • objects, settings, and actions in situations • relatively “local” in time and space • perceived externally • abstract concepts • complex configurations of information distributed across settings and events • internally perceived content especially important • e.g., mental states, affect, cognitive operations • simulating abstract concepts • simulating the associated configuration of information,including mental states and events Schwanenflugel(1991) Grounding Knowledge

Proportions of property types Concept type Entity Setting/Event Mental State Concrete .26 .46 .21 Abstract .15 .52 .28 Exploratory study Barsalou & Wiemer-Hastings (2005) • method • participants produced properties for abstract and concrete concepts • TRUTH, FREEDOM, INVENTION vs. SOFA, BIRD, CAR • results • participants produced broad situational content for both types of concepts • people generally situate both kinds of concepts • abstract concepts activated more mental state and setting/event properties, whereas concrete concepts activated more entity properties • the two kinds of concepts rely on different situational information Grounding Knowledge 28

Assessing simulation in abstract concepts with fMRIWilson, Simmons, Martin, & Barsalou (in preparation) • two phases of the experiment Semantic Priming Phase Localizer Phase • localizer phase • identified brain areas that perform abstract forms of processing • blocked design • priming phase • assessed the semantic content of two abstract concepts • fast event-related design • hypothesis • simulations of abstract processing will represent the abstract concepts Grounding Knowledge 29

Localizer Phase Semantic Priming Phase • thoughts localizer • participants viewed blocks of complex scenes • for each picture, participants answered the following question to themselves • “What are the thoughts of people in the picture?” • counting localizer • participants viewed blocks of complex scenes • for each picture, participants answered the following question to themselves • “How many entities are there in the picture?” Note. Localizer blocks were also included for two concrete localizers, color and motion, not discussed here. Grounding Knowledge 30

Localizer Phase Semantic Priming Phase L x = -45 • thoughts – counting • medial prefrontal, precuneus • bilateral anterior and superior temporal • counting – thoughts • bilateral intraparietal sulcus x = -5 R x = 48 L y = -60 p < .0001, corrected, random effects Grounding Knowledge 31

Localizer Phase Semantic Priming Phase 5 sec (fMRI images of interest) • semantic priming trials • fast-event related design • concepts ordered randomly • random ISI jitter • catch trials to deconvolveword primes and pictures • possible responses: “Word applies” or “Word doesn’t apply” • pictures promoted deepsemantic processing • hypothesis • simulations underlie meaning 2.5 sec convince Response 5 sec (fMRI images of interest) 2.5 sec arithmetic Response Note. Semantic priming trials were also included for two concrete concepts red and rolling, not discussed here. Grounding Knowledge 32

Localizer Phase Semantic Priming Phase L x = -45 • convince – arithmetic • dark blue areas • medial prefrontal, precuneus, superior temporal • no arithmetic – convince activations in localizer areas • simulations underlie meaning • arithmetic – convince • orange areas • intraparietal sulcus • no convince – arithmeticactivations in localizer areas • simulations underlie meaning x = -5 R x = 48 A L y = -60 L y = -45 p < .05, corrected, random effects Grounding Knowledge 33

Assessing simulation in abstract concepts with fMRIWilson, Barrett, Simmons, Barsalou (in preparation) Social Threat Situation You’re leading an important group presentation at work. You’re unprepared for your boss’s questions because a couple of co-workers didn’t pull their weight in preparing for the meeting. The presentation finishes awkwardly. Your boss thanks you coolly. His associates sit looking at each other, wondering what to say next. You can feel the sweat forming under your arms. Physical Threat Situation You row on a lake to experience the feel of storm waves. As you head toward the middle of the lake, white-capped waves break across your small boat with increasing frequency. As the waves become increasingly rough, water pours into the boat. The boat sinks. You try to keep your head above water, as wave after wave crashes over you. Your wet clothes feel heavy on your body, making it difficult to stay afloat. Other concepts used: PLAN FEAR ANGER OBSERVE TASK “How easy was it for you to experience OBSERVE in the context of the situation?” Very easy (3), Somewhat easy (2), Not easy (1) Grounded Cognition 34

physical situations social situations both Observe – (Fear, Anger) Visual processing (bilateral superior occipital) Object processing (bilateral fusiform, BA 37) Auditory processing (bilateral superior temporal) x = -32 x = -47 x = 55 R z = 26 R z = -8 z = 5 R BOLD responses only to words, p < .05, corrected, random effects Grounded Cognition

OBSERVE activations as a function of situation • Physical Social Overlap • 1. L. Sup. Temporal • extends into pole, insula • 2. R. Sup. Temporal • extends into pole • Bilateral Mid Occipital • extends into inf. parietal • 4. L. ITG/MTG • extends into fusiform, PHC • 6. R. ITG/MTG • 7. Precuneus • 8. Mid Cingulate • 9. R. Middle Frontal • 10. R. Insula 8% 83% 9% z = -12 z = 28 y = -17 BOLD responses only to words, p < .05, corrected, random effects Grounded Cognition

Symbolic operations typically viewed as a problem for grounded views assumed to be possible only in amodal symbolic systems examples of symbolic operations predication “Pumpkins are orange” ORANGE (pumpkins) conceptual combination “The cat is on the sofa”  ON (cat, sofa) simulation-based accounts of symbolic operations Barsalou, L.W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577-609. Barsalou, L.W. (2003). Abstraction in perceptual symbol systems. Philosophical Transactions of the Royal Society of London: Biological Sciences, 358, 1177-1187. Barsalou, L.W. (2008). Grounding symbolic operations in the brain’s modal systems. In G.R. Semin & E.R. Smith (Eds.), Embodied grounding: Social, cognitive, affective, and neuroscientific approaches (pp. 9-42). New York: Cambridge University Press. Grounding Knowledge

Grounding color semantics in the visual systemSimmons, Ramjee, Beauchamp, McRae, Martin, & Barsalou (2007) WATER . . white . . (false) MILK . . white . . (true) • participants verified “possible” properties of conceptsin an fMRI scanner • color properties • motor properties • hypothesis • verifying color properties produces color simulations in the brain’s color perception system • builds on previous findings in the literature • e.g., Chao & Martin (1999), Oliver & Thompson-Schill (2003) DOOR KNOB . . turned . . (true) • 10 participants, event-related design (168 trials across 7 runs) • concept-only catch trials used to deconvolve concepts and properties Grounding Knowledge

Color localizer task Establishing the color perception areas • the Farnsworth-Munsell color judgment task (adapted for fMRI) • participants judged whether or not hues are ordered from lightest to darkest • for multiple chromatic and achromatic wheels • performed after the property verification scanning runs Achromatic Chromatic PropertyVerification Localizer Ordered Not ordered • blocked design, 4 runs • 3 chromatic blocks / run • 3 achromatic blocks / run Grounding Knowledge

Areas active for color perception Front Chromatic > Achromatic Wheels PropertyVerification Localizer p < .05, corrected, random effects Left Grounding Knowledge

Areas active for verifying color properties Front PropertyVerification Localizer Color > Motor Properties p < .05, corrected, random effects Left Grounding Knowledge

Overlapping activations for verifying color properties and perceiving color Front Overlap Color > Motor Properties Chromatic > Achromatic Wheels Tootell et al. (2004) argue that this overlapping area is a primary color processing area in macaques A subsequent ROI analysis on the fusiform cluster identified by the color perception task (-33, -36, -9)found color > motor properties, p < .05, corrected, random effects. Left Grounding Knowledge

Occlusion effects in conceptual combinationWu & Barsalou (2009) • in perception, occluded features are less salient than unoccluded ones • for LAWN, dirt and roots are less salient than green and blades • in conceptual processing, is there an analogous occlusion effect? • if simulation is used to combine meanings, there should be • nouns task • some participants generated features for nouns having occluded features • neutral instructions (nothing mentioned about imagery) • simulation prediction: • unoccluded features should be produced more than occluded features LAWN  green, blades > dirt, roots • noun phrases (NPs) task • other participants generated features for NPs with revealing modifiers • either novel or familiar NPs (neutral instructions) • simulation prediction: • occluded features should be produced more often than for isolated nouns ROLLED-UP LAWN  dirt, roots > LAWN  dirt, roots Grounding Knowledge 46

Results • occlusion affected conceptual processing • external features more likely than internal features for nouns • internal features become more likely for both novel and familiar NPs • not the result of rules associated with modifiers • e.g., occluded features are not produced more often for ROLLED UP SNAKE than for SNAKE Grounding Knowledge 47

Assessing conceptual combination with fMRIJames, Simmons, Barbey, Hu, & Barsalou (in preparation) two critical kind of trials independent " . . " combination " − . " fast event-related design trial types interspersed randomly random ISI jitter catch trials to deconvolvemodifiers and head nouns familiarity responses at " . " “Occurs once a month or more” or “Occurs less than once a month” Independent Trials Modifier 1 sec Head Noun 1 sec reverend . distressed . 3 sec 3 sec Judge Modifier Judge Head Noun Familiarity Familiarity Combination Trials Modifier 1 sec Head Noun 1 sec reverend . distressed − 3 sec 3 sec Judge Noun Phrase Familiarity Grounding Knowledge

Examples of modifiers and head nouns Independent Trials • mental state modifiers distressed reverend pleasing cloves • motion modifiers soaring balloon swaying oak • location modifiers ocean shrimp auditorium piano • no modifier or head nounrepeated • head nouns counter-balanced forlength, frequency, category, typicality • words counter-balanced across participants so that every word occurred in both the independent and combination conditions Modifier 1 sec Head Noun 1 sec reverend . distressed . 3 sec 3 sec Judge Modifier Judge Head Noun Familiarity Familiarity Combination Trials Modifier 1 sec Head Noun 1 sec reverend . distressed − 3 sec 3 sec Judge Noun Phrase Familiarity Grounding Knowledge

Mental State modifiers – Motion and Location Modifiers Motion modifiers – Mental State and Location Modifiers Location modifiers – Mental State and Motion Modifiers Grounding Knowledge

Grounding Knowledge in the Brain’s Modal Systems