Situation Models and Embodied Language Processes

1. Situation Models and Embodied Language Processes Franz Schmalhofer University of Osnabrück / Germany • Memory and Situation Models • Computational Modeling of Inferences • What Memory and Language are for • Neural Correlates • Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

2. Prior to the twentieth century: knowledge was assumed to be perceptual Past several decades: fields of cognition and perception have diverged. Perceptual approaches viewed as untenable for conceptual systems. Logic, statistics, programming languages have inspired amodal theories different from perceptional characteristics Theories of Knowledge

3. Symbol Grounding Illustration for computational theories of language understanding: The Chinese room (Searle, 1980) - getting Chinese input symbols - manipulation of symbols only according to their shapes ( no meanings/no „understanding“ ) • returning Chinese symbols as output The symbol grounding problem (Harnad, 1990) cognition cannot be just symbol manipulation

4. Perceptual Symbol Systems (Barsalou, 1999) • WRONG: Perceptual systems pick up information from the environment and pass it on to separate systems that support the various cognitive functions. (i.e. language, memory, and thought) • CORRECT: Cognition is inherently perceptual, sharing systems with perception at both the cognitive and the neural levels. • No divergence between cognition and perception

5. How we got the wrong ideas • Behaviorist attacks on mentalism. (Watson) • Similar attacks from ordinary language philosophy (Wittgenstein) • Continuing attacks after the cognitive revolution. • Development of logic, programming languages, statistical representation • Contributions of amodal symbol systems : • Formalizable, runnable, applicable • Highlight important computational properties of productivity, proposition, structure, etc.

6. Assumption: Cognition is Grounded in Perception • A common representational system underlies perception and cognition. • From Aristoteles to Locke to Kant, theorists over the last 2000 years have viewed cognition as imagistic in nature. • Image-based theories disappeared as behaviorists and language philosophers began to avoided to talk about mental states.

7. How is cognition grounded in perception? • During perceptual experience, association areas in the brain capture bottom-up patterns of activation in sensory-motor areas and in a top-down manner, association areas partially reactivate sensory-motor areas to implement perceptual symbols. • The storage and reactivation of perceptual symbols operates at the level of perceptual components--not at the level of holistic perceptual experiences. • Use of selective attention, schematic representations of perceptual components are extracted from experience and stored in memory

8. Properties of amodal symbol system • As amodal symbols are transduced from perceptual states, they enter into larger representational structures. • In turn, these structures support all of the higher cognitive functions, including knowledge, memory, language and thought. • Across the cognitive sciences, standard theories of knowledge adopt the assumptions, so called amodal symbol systems. • Information in the physical world produces neural states in perceptual systems. • A transduction process takes these states as input, produces descriptions of them in a completely new representation language.

9. 4. Amodal symbol system Amodal symbol representations • Perceptual states are transduced into a completely new representational system that describes these states amodally. • The internal structure of these symbols is unrelated to the perceptual state that produced them.

10. Problems with amodal symbol system • No account for the relation between cognition and perception. • No empirical evidence that amodal systems exist. • Transduction process that maps perceptual states into amodal symbols remains unclear. • Converse of transduction problem : no account how perceptual states map to amodal symbols. • Too powerful : amodal symbol systems are unconstrained, offer little insight into the phenomena.

11. Resurgence of perceptual symbol system • Theorists develop perceptual views that are provocative, powerful. • It provides a natural account of the relation between cognition and perception. • An obvious account exists of how perceptual symbols are implemented in the brain. • No need for a major leap in evolution • Provide natural mechanisms for representing space and time • Make clear a priori predictions that are falsifiable • Growing empirical evidence for perceptual symbols

12. Basic assumption of perceptual views • States arise in sensory-motor systems during contact with the physical world. • traditionally viewed as conscious states • but will be viewed here primarily as neural states • these sensory-motor states are stored in memory to some extent (utilizing sensory-motor systems) • stored perceptual states later support higher cognitive processes • during memory, language, and thought • may establish reference back into the physical world

13. Representation in Perceptual Symbol Systems • Subsets of perceptual states in sensory-motor systems are extracted. • The internal structure of these symbols is therefore • modal and • analogically related to the perceptual state that produced them.

15. Core assumptions about perceptual symbols • Perceptual symbols are schematic. • Perceptual symbols are multimodal • Perceptual symbols enter into simulation competence. • Perceptual symbols are productive. • Perceptual symbols represent situation components. • Perceptual symbols also represent abstract concepts

16. Experiments • Spivey, M.J. et al. (2000) Eye movements during comprehension of spoken scene descriptions. • Zwaan, R.A., Stanfield, R. Y. & Yaxley, R. H. (2002) Do language comprehenders routinely represent the shapes of objects. Psychological Science, 13, 168-171. • Glenberg A. M. & Kaschak M. P. (2002) Grounding language in action. Psychonomic Bulletin & Review, 9, 558-565

17. Zwaan (2004) The immersed experiencer C = construal T = time S = spatial region (personal, action, vista) P = perspective F = focal entity R = relation B = background entity f = feature

18. Embodied Language Comprehension • Taylor and Tversky (1992): “language is a surrogate for experience” • Goal of language comprehension: creation of an embodied mental model • In the brain words activate experiences with their referents (Pulvermüller) • Perceptual simulation of a described object or situation; construction of a situation model

19. Visual field • Fovea • Para-fovea • Plateau • Periphery • Temporal monocular

20. From the Retina to V1 nasal retinal fibres temporal retinal fibres

21. Visual Pathways ‘Vision for Action’ ‘Vision for Perception’ Goodale & Humphrey (1998)

22. Perceptual experiences activate association areas in the brain which capture bottom-up patterns of activation in sensory-motor areas. top-down patterns, association areas partially reactivate sensory-motor areas to implement perceptual symbols. Perceptual States: Arise in sensory-motor systems with two components Unconscious neural representation of physical input Optional conscious experience Related perceptual symbols combine to form simulators that allow “the cognitive system to construct specific simulations of an entity or event in its absence Perceptual Experience and Perceptual States

23. Perceptual symbols • Are patterns that rise in hierarchical feature maps of sensory-motor systems during perception and action. • May or may not be topographical;captured by association areas (Damasio, 1989) • From local, to poly-sensory and to frontal • Tuned for specific combinations of features • Activating an associative pattern reinstates • not necessarily complete nor veridical but partial records of the neural states that underlie perception • Is dynamic, not discrete • not necessarily representative of specific individuals • potentially indeterminate

25. Perceptual symbols are NOT • like physical pictures • entire perceptual states • If they were, the componential character of conceptual system would not be feasible • mental images or conscious experience • states in neural systems

26. Simulators • Perceptual symbols of the same category are integrated together in a single system (simulators). • A simulator • An organized system of perceptual symbols that can produce simulations of a category in the absence of physical exemplars. • Typically contains perceptual symbols extracted from many members of category on many modalities. • Composed of frames, the simulations that the frame produces.

27. Simulators and Simulations • Specific runs of a simulator that reenact the multimodal experience of a category • Utilize a small subset of the information in a simulator • Infinite many simulations is possible. • Simulations are always partial and sketchy, never complete. • A simulator goes beyond a simple empirical collection of sense impressions. • A huge set of simulations that include the range of experience associated with a category • i.e. the representation of a type, not a token

29. Simulations as vehicles. This figure is showing some partial frame for car, it illustrates how the frame has changed dynamically.

30. Categorization by construal • When an entity is categorized, the best fitting simulator is found • The simulator runs a simulation that provides a good fit to the entity • Some examples • Hearing a bark and simulating a dog • Seeing a growling dog and simulating the experience of being attacked • Smelling food and simulating what it is

31. Concepts and Offline Conceptualization • Concepts and Simulators • A concept is equivalent to a simulator. • If we have an appropriate simulator of something, then it can be said that we understand the concept. • Goal of human learning is to establish simulators. • Offline conceptualizing during memory, language and thought • Simulations provide inferences about likely properties of entities in their absence • The simulations run in sensory-motor systems and they can be mapped later to perceived entities • E.g. remembering one’s parking spot, finding the referent of a linguistic description.

33. Summary I: Perceptual Symbol Systems • Perceptual states arise in the sensory-motor system. • A subset of the state is extracted by selective attention and stored in long-term memory. • This perceptual memory can function as a symbol entering into symbol manipulation. • Collections of the perceptual symbols comprise our conceptual representations. • The structure of a perceptual symbol corresponds (at least somewhat) to the perceptual state that produced it. • Note: this does not claim that a perceptual symbol corresponds to the physical world.

34. Summary II: Perceptual Symbol Systems • A very different approach to knowledge in the form of perceptual symbol systems. • Perceptual states are not transduced into a completely new representational language, instead subsets of perceptual states are extracted to function symbolically and support the higher cognitive functions. • Reference: Barsalou, L. W. (1999). “Perceptual Symbol Systems.” Behavioral and Brain Sciences 22: (507-569).