Embodiment and computation convergent constraints on language use
This presentation is the property of its rightful owner.
Sponsored Links
1 / 75

Embodiment and Computation: Convergent Constraints on Language Use PowerPoint PPT Presentation


  • 44 Views
  • Uploaded on
  • Presentation posted in: General

Embodiment and Computation: Convergent Constraints on Language Use. Nancy Chang [email protected] UC Berkeley / International Computer Science Institute. “ Harry walked to the cafe.”. “Harry walked into the cafe.”. CAFE. CAFE. What does language do?.

Download Presentation

Embodiment and Computation: Convergent Constraints on Language Use

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Embodiment and computation convergent constraints on language use

Embodiment and Computation:Convergent Constraints on Language Use

Nancy Chang

[email protected]

UC Berkeley / International Computer Science Institute


What does language do

“Harry walked to the cafe.”

“Harry walked into the cafe.”

CAFE

CAFE

What does language do?

A sentence can evoke an imagined scene and resulting inferences:

  • Goal of action = atcafe

  • Source = awayfrom cafe

  • cafe = point-like location

  • Goal of action = insidecafe

  • Source = outsidecafe

  • cafe = containing location


Embodied inferences

The scientist walked into the wall.

WALL

Bonk!!

Embodied inferences

The hobo drifted into the house.

Thesmoke drifted into the house.


Metaphorical inference

Metaphorical inference

  • France fell into recession. Germany pulled it out.

  • The economy is moving along at the pace of a Clinton jog.

  • The Indian Government is stumbling in implementing its liberalization plan.


Embodied knowledge needed

Embodied knowledge needed

  • What things can serve as containers?

    • rooms but not walls (usually)

  • How do different entities interact?

    • how people and gases interact with houses.

  • How are different actions/states related?

    • stumbling / walking, falling / containment

  • How can actions vary?

    • rate, direction, degree of force, etc.

… that is, more than predicate-argument structure!

WALK(x), FALL(y), HIT(x,y), etc.


Embodiment in language

Embodiment in language

  • Perceptual and motor systems play a central role in language production and comprehension

  • Theoretical proposals

    • Linguistics: Lakoff, Langacker, Talmy

    • Neuroscience: Damasio, Edelman

    • Cognitive psychology: Barsalou, Gibbs, Glenberg, MacWhinney

    • Computer science: Steels, Feldman


Goal computationally precise theories of language

Goal: computationally precise theories of language

Theory of Language Structure

Theory of Language Acquisition

Theory of Language Use


Embodiment and computation convergent constraints on language use

Theory of Language Structure

Theory of Language Acquisition

Theory of Language Use


Embodiment and computation convergent constraints on language use

Theory of Language Structure

Theory of Language Acquisition

Theory of Language Use


Simulation hypothesis

Simulation hypothesis

We understand utterances by mentally simulating their content.

  • Simulation exploits some of the same neural structures activated during performance, perception, imagining, memory…

  • Linguistic structure parametrizes the simulation.

    • Language gives us enough information to simulate


Language understanding as simulative inference

Utterance

“Harry walked to the cafe.”

Linguisticknowledge

Analysis Process

General Knowledge

Simulation Specification

SchemaTrajectorGoal

walkHarrycafe

Belief State

Cafe

Simulation

Language understanding as simulative inference


Embodiment and computation convergent constraints on language use

  • Embodiment and Simulation

“What is an idea?

It is an image that paints itself in my brain.”

— Voltaire


Computational efficacy

Computational efficacy

  • Embodied representations the norm in robotics!

  • Computational representations for lexical semantics: have been developed for:

    • Spatial relations (Regier 1996)

    • Actions (Bailey 1997, Narayanan 1997)

    • Objects / attributes (Roy 1998)

  • Metaphor understanding system based on simulation (Narayanan 1997)


Metaphor system architecture

Metaphor system architecture

Target domain

Metaphor maps

Source domain

(Narayanan 1997)


Metaphor understanding system

Indian Government stumbling in implementing liberalization plan

Metaphor understanding system


Missing link grammar

Missing link: grammar!

  • Metaphor understanding system demonstrates that embodied inferences for difficult case are feasible.

    • BUT: system has no grammar!

    • How do we bridge the gap?

  • Need a grammatical theory/formalism that can served as an interface between linguistic units and embodied, dynamic, encyclopedic, context-based information (i.e., that can support simulation).


Embodiment and computation convergent constraints on language use

2. Embodied Construction Grammar

”It is not enough to say that the mind is embodied; one must say how.”— Damasio


What passes as grammar

“Syntactic investigation of a given language has as its goal the construction of a grammar that can be viewed as a device of some sort for producing the sentences of the language under analysis.”(Chomsky 1957)

Inadequate notion of grammar

Meaning-free: syntax separate from meaning, function and processing; unanalyzable symbolic units

Inflexible: strict word order, strictly hierarchical, strictly compositional

What passes as grammar?


Who s up to the task

Who’s up to the task?

  • Most theories of language are not explicitly and systematically tied to action and perception

  • Promising exceptions

    • Cognitive Grammar / cognitive linguistics

    • Construction Grammar

    • Typically criticized for being informal / vague

  • We borrow liberally from both and formalize.


Cognitive linguistics

Cognitive Linguistics

“Language is an integral part of cognition which reflects the interaction of cultural, psychological, communicative, and functional considerations, and which can only be understood in the context of a realistic view of conceptualization and mental processing.”

International Cognitive Linguistics Association website(http://www.cognitivelinguistics.org/aims.shtml)


Key borrowed ideas

Key borrowed ideas

  • Conceptual structures are embodied.

    • Meaning is conceptualization (part of larger cognitive system).

    • Concepts are grounded in human experience as physical, psychological and social beings in the world.

  • Basic symbolic unit at all levels is a form-meaning pair, or construction.

    • Syntax is not independent of semantics.

    • Phrasal/clausal constructions can contribute meaning independently of constituents.

(Lakoff 1987, 1985; Langacker 1991, 1987)

(Fillmore 1988, Kay & Fillmore 1999, Lakoff 1987, Goldberg 1995)


Traditional levels of analysis

Traditional levels of analysis

UTTERANCE

Pragmatics

Semantics

Syntax

Morphology

Phonology

Phonetics


Form meaning mappings for language

Form

phonological cues

word order

intonation

inflection

Meaning

event structure

sensorimotor control

attention/perspective

social goals...

Cafe

Form-meaning mappings for language

Linguistic knowledge consists of form-meaning mappings:


Construction grammar

Trajector

Source

Goal

Path

Construction Grammar

A construction is a form-meaning pair whose properties may not be strictly predictable from other constructions.

(Construction Grammar, Goldberg 1995)

Form

Meaning

block

walk

to


Embodiment and computation convergent constraints on language use

Constructions as maps between relations

Complex constructions are mappings between relations in form and relations in meaning.

Form

Meaning

  • Mover + Motion

    • before(Mover, Motion)

MotionEventmover(Motion, Mover)

“is” + Action+ “ing”before(“is”, Action)suffix(Action, “ing”)

ProgressiveActionaspect(Action, ongoing)

DirectedMotionEventdirection(Motion, Direction)mover(Motion, Mover)

Mover+Motion+Directionbefore(Motion, Direction)before(Mover, Motion)


More on construction grammar

More on Construction Grammar

(Goldberg 1995)

  • Clause-level patterns correspond to basic events

    transitive: Agent Action Patient

    ditransitive (dative): Giver Action Recipient Gift

  • Economical: no explosion of senses

    Hepushedthe ball.

    He pushed her the ball.

  • Novel uses handled more robustly

    Mary pushedthe tissue off the table.

    ?Mary sneezedthe tissue off the table.

    *Mary sleptthe tissue off the table.


Embodied construction grammar bergen and chang 2002

Embodied Construction Grammar(Bergen and Chang 2002)

  • Embodied representations

    • active perceptual and motor schemas(image schemas, x-schemas, frames, etc.)

    • situational and discourse context

  • Construction Grammar

    • Linguistic units relate form and meaning.

    • Both constituency and (lexical) dependencies allowed.

  • Constraint-based

    • based on feature structure unification (as in HPSG)

    • Diverse factors can flexibly interact.


Ecg structures

ECG Structures

  • Schemas

    • image schemas, force-dynamic schemas, executing schemas, frames…

  • Constructions

    • lexical, grammatical, morphological, gestural…

  • Maps

    • metaphor, metonymy, mental space maps…

  • Spaces

    • discourse, hypothetical, counterfactual…


Image schemas

boundary

bounded region

Image schemas

  • Trajector / Landmark (asymmetric)

    • The bike is near the house

    • ? The house is near the bike

  • Boundary / Bounded Region

    • a bounded region has a closed boundary

  • Topological Relations

    • Separation, Contact, Overlap, Inclusion, Surround

  • Orientation

    • Vertical (up/down), Horizontal (left/right, front/back)

    • Absolute (E, S, W, N)

LM

TR


Embodied schemas

Embodied schemas

schema name

schemaSource-Path-Goal

roles

source

path

goal

trajector

schemaContainer

roles

interior

exterior

portal

boundary

role name

Boundary

Interior

Trajector

Portal

Source

Goal

Path

Exterior

These are abstractions over sensorimotor experiences.


Embodied constructions

CAFE

Embodied constructions

ECG Notation

Form

Meaning

constructionHARRY

form : /hEriy/

meaning : Harry

Harry

constructionCAFE

form : /khaefej/

meaning : Cafe

cafe

Constructions have form and meaning poles that are subject to type constraints.


Representing constructions t o

Representing constructions: TO

constructionTO

form

selff.phon /thuw/

meaning

evokes

Trajector-Landmark as tl

Source-Path-Goal as spg

constraints:

tl.trajector«spg.trajector

tl.landmark«spg.goal

local alias

identification constraint

The meaning pole may evoke schemas (e.g., image schemas) with a local alias. The meaning pole may include constraints on the schemas (e.g., identification constraints «).


The i nto construction

The INTO construction

constructionINTO

form

selff.phon /Inthuw/

meaning

evokes

Trajector-Landmark as tl

Source-Path-Goal as spg

Container as cont

constraints:

tl.trajector«spg.trajector

tl.landmark«cont

cont.interior«spg.goal

cont.exterior«spg.source

  • TO vs. INTO:

    • INTO adds a Container schema and appropriate bindings.


Constructions with constituents the s patial p hrase construction

Constructions with constituents:The SPATIAL-PHRASEconstruction

constructionSPATIAL-PHRASE

constructional

constituents

sp : Trajector-Landmark

lm : Thing

form

spfbefore lmf

meaning

spm.landmark « lmm

local alias

order constraint

identification constraint

  • Constructions may alsospecify constructional constituents and impose form and meaning constraints on them:

    • order constraints

    • identification constraints


An argument structure construction

An argument structure construction

constructionDIRECTED-MOTION

subcase of Pred-Expr

constructional

constituents

a : Ref-Exp

m: Pred-Exp

p : Spatial-Phrase

form

afbefore mf

mfbefore pf

meaning

evokes Directed-Motion as dm

selfm.scene « dm

dm.agent « am

dm.motion «mm

dm.path «pm

schemaDirected-Motion

roles

agent : Entity

motion : Motion

path : SPG


The c aused m otion construction

The CAUSED-MOTIONconstruction

constructionCAUSED-MOTION

subcase of Pred-Expr

constructional

constituents

agent : Entity

action: Action

patient: Entity

path : SPG

form

agentfbefore actionf

actionfbefore patientf

actionfbefore pathf

meaning

evokes Caused-Motion as cm

selfm.scene « cm

cm.agent « agentm

cm.action «actionm

cm.patient«patientm

cm.path «pathm


Language understanding process

Language Understanding Process

  • An utterance is perceived

  • This activates the form pole of some constructions

  • The Analysis process assembles the constructions, using construal where necessary, and binds together their forms and their meanings

  • The product is a constructional analysis

  • This yields a semspec -- parameterized schemas linked together in specified ways

  • The semspec is input into the Simulation process, where the understander imagines the content

  • Resulting inferences are propagated through the conceptual system.


Simulation based language understanding

constructionWALKED

form

selff.phon [wakt]

meaning : Walk-Action

constraints

selfm.time before Context.speech-time

selfm..aspect  encapsulated

“Harry walked into the cafe.”

Utterance

Analysis Process

Constructions

General Knowledge

Semantic

Specification

Belief State

Simulation

CAFE

Simulation-based language understanding


Simulation specification

Simulation specification

  • A simulation specification consists of:

  • schemas evoked by constructions

  • bindings between schemas


Language understanding process1

Language Understanding Process


Constructional analysis

Constructional analysis


Semantic specification

Semantic Specification


Basic feature structure

Basic Feature Structure

A new rule for “I”

The corresponding fstruct

Pronoun  I

number SG

person  1st

-The top part of the rule is the old

CFG rule.

-This data structure is attached to the

nonterminal during parsing so that the

parser can use the information.

-The next two lines set the agreement

features.

-The feature is on the lhs of the colon

And the value is rhs of the colon.

-The  denotes assignment to the

feature listed on the lhs.


Feature structure unification

Feature Structure Unification

  • To check the compatibility of two fstructs

    • Two feature structures are compatible if they have the same value for every feature they have in common (or if one or both leave the value unspecified).

    • This process of checking compatibility is called unification.

  • Unification

    • Is a recursive process that takes two feature structures and either returns the combined feature structure if they are compatible or it returns failure.

    • Base case: Two values unify if they are the same string.

    • Recursive Case: Two feature structures unify if for each feature they have in common, those values unify.

    • The resulting feature structure just adds the features they don’t have in common to the resulting structure.


Language analysis and embodied construction grammar

Language Analysis and Embodied Construction Grammar

John Bryant

[email protected]


Getting from the utterance to the semspec

Getting From the Utterance to the SemSpec

  • Need a grammar formalism

    • Embodied Construction Grammar (Bergen & Chang 2002)

  • Need new models for language analysis

    • Traditional methods too limited

    • Traditional methods also don’t get enough leverage out of the semantics.


Embodied construction grammar

Embodied Construction Grammar

  • Semantic Freedom

    • Designed to be symbiotic with cognitive approaches to meaning

    • More expressive semantic operators than traditional grammar formalisms

  • Form Freedom

    • Free word order, over-lapping constituency

  • Precise enough to be implemented


Traditional parsing methods fall short

Traditional Parsing Methods Fall Short

  • PSG parsers too strict

    • Constructions not allowed to leave constituent order unspecified

  • Traditional way of dealing with incomplete analyses is ad-hoc

    • Making sense of incomplete analyses is important when an application must deal with “ill-formed” input

  • Traditional unification grammar can’t handle ECG’s deep semantic operators.


Recognizer example

Recognizer Example

Mary kicked the ball into the net.

This is the initial Constituent Graph for caused-motion.

Patient

Agent

Action

Path


Recognizer example1

Recognizer Example

Construct:

Caused-Motion

Constituent:

Agent

Constituent:

Action

Constituent:

Patient

Constituent:

Path

The initial constructional tree for the instance of

Caused-Motion that we are trying to create.


Recognizer example2

Recognizer Example


Recognizer example3

Recognizer Example

processed

Mary kicked the ball into the net.

A node filled with gray is removed.

Patient

Agent

Action

Path


Recognizer example4

Recognizer Example

Construct:

Caused-Motion

RefExp:

Mary

Constituent:

Action

Constituent:

Patient

Constituent:

Path

Mary kicked the ball into the net.


Recognizer example5

Recognizer Example


Recognizer example6

Recognizer Example

processed

Mary kicked the ball into the net.

Patient

Agent

Action

Path


Recognizer example7

Recognizer Example

Construct:

Caused-Motion

RefExp:

Mary

Verb:

kicked

Constituent:

Patient

Constituent:

Path

Mary kicked the ball into the net.


Recognizer example8

Recognizer Example


Recognizer example9

Recognizer Example

processed

Mary kicked the ball into the net.

According to the Constituent Graph,

The next constituent can either be the

Patient or the Path.

Patient

Agent

Action

Path


Recognizer example10

Recognizer Example

processed

Mary kicked the ball into the net.

Patient

Agent

Action

Path


Recognizer example11

Recognizer Example

Construct:

Caused-Motion

RefExp:

Mary

Verb:

kicked

RefExp:

Det Noun

Constituent:

Path

Det

Noun

Mary kicked the ball into the net.


Recognizer example12

Recognizer Example


Recognizer example13

Recognizer Example

processed

Mary kicked the ball into the net.

Patient

Agent

Action

Path


Recognizer example14

Recognizer Example

Construct:

Caused-Motion

RefExp:

Mary

Verb:

kicked

RefExp:

Det Noun

Spatial-Pred:

Prep RefExp

RefExp

Det

Noun

Prep

Det

Noun

Mary kicked the ball into the net.


Recognizer example15

Recognizer Example


Resulting semspec

Resulting SemSpec

After analyzing the sentence, the following identities are asserted in the resulting SemSpec:

Scene = Caused-Motion

Agent = Mary

Action = Kick

Patient = Path.Trajector = The Ball

Path = Into the net

Path.Goal = The net


Summary

Summary

  • By expanding traditional notions of parsing and unification grammar, it is possible to make a robust ECG-based language analyzer.

  • Further work is necessary to better ground partial analysis/semantic density.

    • But they seem promising.


Embodiment and computation convergent constraints on language use

Embodied Construction Grammar providesformal tools for linguistic description and analysis motivated largely by cognitive/functional concerns.

  • A shared theory and formalism for different cognitive mechanisms

    • Constructions, metaphor, mental spaces, etc.

  • Precise specifications of structures/processes involved in language understanding

  • Bridge to detailed simulative inference using embodied representations


Summary ecg

Summary: ECG

  • Linguistic constructions are tied to a model of simulated action and perception

  • Embedded in a theory of language processing

    • Constrains theory to be usable

    • Frees structures to be just structures, used in processing

  • Precise, computationally usable formalism

    • Practical computational applications, like MT and NLU

    • Testing of functionality, e.g. language learning

  • A shared theory and formalism for different cognitive mechanisms

    • Constructions, metaphor, mental spaces, etc.


Ecg applications

ECG applications

  • Grammar

    • Spatial relations/events(Bergen & Chang 1999; Bretones et al. In press)

    • Verbal morphology(Gurevich 2003, Bergen ms.)

    • Reference: measure phrases(Dodge and Wright 2002), construal resolution(Porzel & Bryant 2003), reflexive pronouns(Sanders 2003)

  • Semantic representations / inference

    • Aspectual inference(Narayanan 1997; Chang, Gildea & Narayanan 1998)

    • Perspective / frames(Chang, Narayanan & Petruck 2002)

    • Metaphorical inference(Narayanan 1997, 1999)

    • Simulation semantics(Narayanan 1997, 1999)

  • Language acquisition

    • Lexical acquisition(Regier 1996, Bailey 1997)

    • Multi-word constructions(Chang 2004; Chang & Maia 2001)


Embodiment and computation convergent constraints on language use

3. Simulation-based inference


Interpretation simulation

Interpretation: simulation

Constructions can:

  • specify which schemas and entities are involved in an event, and how they are related

  • profile particular stages of an event

  • set parameters of an event

walker at goal

energy

goal=home

walker=Harry

Harryiswalkinghome.


Simulation semantics

Simulation Semantics

  • execution-based model of events/processes

    • tractable, distributed, concurrent, context-sensitive

  • X-schemas provide natural model of

    • resource consumption/production

    • goals, preconditions, effects

    • hierarchical events (multiple granularities)


Simulation semantics 2

Simulation Semantics (2)

  • Captures fine-grained distinctions needed for interpretation

    • aspectual inferences [Narayanan 1997, 1999; Chang et al. 1998]

    • metaphoric inferences [Narayanan 1997, 1999]

    • perspectival inferences [Chang et al. 2002]

    • inductive bias for language learning [Bailey 1997, Chang 2000]

  • Captures essential features of neural computation [Feldman & Ballard 1982, Feldman 1989, Valiant 1994]

    • active, context-sensitive knowledge representation

    • same representational substrate for action, perception[Boccino et al. 2001, NBL01, CNS02]

    • natural model of concurrent and distributed computation


Simulation semantics1

Simulation Semantics

  • Inspired by biological control theory, Simulation Semantics models events as executing-, or x-schemas.

  • An x-schema is a Petri net: a weighted graph consisting of places (circles) and transitions (rectangles) connected by directed input and output arcs.

  • A state is defined by the placement of a token (a black dot or number) in a particular place.

  • The real-time execution semantics of Petri nets models the production and consumption of resources:

    • A transition is enabled when its input places are marked such that it can fire by movement of tokens from input to output.

    • Arcs include resource, enable and inhibitory arcs.

    • Actions have hierarchical structure, permitting embeddings.


Embodiment and computation convergent constraints on language use

Language is embodied:it is learned and used by people with bodies who inhabit a physical, psychological and social world.


  • Login