incremental references in dialogue data from corpora questions for psychologists l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Incremental references in dialogue: data from corpora, questions for psychologists PowerPoint Presentation
Download Presentation
Incremental references in dialogue: data from corpora, questions for psychologists

Loading in 2 Seconds...

play fullscreen
1 / 49

Incremental references in dialogue: data from corpora, questions for psychologists - PowerPoint PPT Presentation


  • 301 Views
  • Uploaded on

Incremental references in dialogue: data from corpora, questions for psychologists Massimo Poesio (Uni Essex) Leverhulme Network opening workshop Edinburgh, 14 th August 2005 Network subprojects involving Essex

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Incremental references in dialogue: data from corpora, questions for psychologists' - andrew


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
incremental references in dialogue data from corpora questions for psychologists

Incremental references in dialogue: data from corpora, questions for psychologists

Massimo Poesio (Uni Essex)Leverhulme Network opening workshop

Edinburgh, 14th August 2005

network subprojects involving essex
Network subprojects involving Essex
  • Incremental interpretation in dialogue, focusing especially on interpretation of anaphoric expressions (Question (1))
  • Alignment and related phenomena: misunderstandings, completions (Question (2))
  • Visual attention and reference (Question (3))
the trains data i incremental interpretation of references
The TRAINS data, IIncremental interpretation of references

M: so we should move the engine at Avon,engine E, to …S: engine E1M: E1S: okayM: engine E1, to Bath, to / or, we could actually move it to Dansville to pick up the boxcar there

the trains data ii shifting the focus of visual attention
The TRAINS data, IIShifting the focus of (visual) attention

M: while this is happening, take engine E1 to Dansville, pick up the boxcar, and come back to AvonS: okayM: okay then load the boxcar with bananas

completions and incremental reference in the bielefeld toyplane corpus
Completions and incremental reference in the Bielefeld ToyPlane corpus

Inst: So, jetzt nimmst DuWell, now you grasp

Cnst: eine Schraubea screw.

Inst: eine <-> orangene mit einem Schlitz.an <-> orange one with a slit

Cnst: Ja. Inst: Und steckst sie dadurch, also And you put it through, let’s see ….Cnst: von oben

the theoretical model resulting from trains ptt
The theoretical modelresulting from TRAINS : PTT
  • Key characteristics:
    • Building on (Compositional) DRT (Muskens, 1996)
    • Common ground as a record of the discourse situation (Barwise and Perry, 1983)
    • An account of incremental semantic interpretation
    • (An account of GROUNDING and alignment: last year’s Bielefeld / CATALOG presentations)
  • Other people involved in the development:
    • David Traum, Hannes Rieser, Colin Matheson, Reinhard Muskens
from drt to ptt
From DRT to PTT
  • a. A: There is an engine at Avon.
  • B: It is hooked to a boxcar
  • DRT: [ x,w,y,u,s,s’| engine(x), Avon(w), s: at(x,w), boxcar(y), s’:hooked-to(u,y), u=x]
common ground and discourse situation in ptt
Common ground and discourse situation in PTT

[ce1,ce2,K1,K2|

K1=[x,w,s| engine(x), Avon(w), s: at(x,w)],

ce1: assert(A,B,K1)

K2=[y,z,s’| boxcar(y), s’:hooked-to(z,y), z=x],

ce2: assert(B,A,K2)]

utterances in the discourse situation
Utterances in the discourse situation

[u1,u2, ce1,ce2,K1,K2|

u1: utter(A,”there is an engine at Avon”),K1=[x,w,s| engine(x), Avon(w), s: at(x,w)],ce1: assert(A,B,K1),generate(u1,ce1),u2: utter(B,”it is hooked to a boxcar”), K2=[y,z,s’| boxcar(y), s’:hooked-to(z,y), z=x], ce2: assert(B,A,K2),generate(u2,ce2) ]

incremental interpretation in ptt
Incremental interpretation in PTT
  • (Poesio, 1991, 1992, 1994, 1995, 1999): each perceptual event leads to an update of the discourse situation
    • Sub-sentential (and sub-word) utterances
    • Perception of non-verbal events (e.g., gestures), attention shift
  • Following each update, the discourse situation is further updated via non-monotonic inference
    • (formulated in terms of Prioritized Default Logic, Brewka 1994)
micro conversational events poesio 1995
Micro conversational events (Poesio, 1995)

boxcar  [u|u:utter(A,”boxcar”), Noun(u),sem(u)=x [|boxcar(x)], + SYN INFO (NEXT)]

umm [u,ce| u: utter(A,”umm”), ce: keep-turn(A), generate(u,ce)]

mces in the example from the bielefeld corpus
MCEs in the example from the Bielefeld corpus

Inst: So, jetzt nimmst DuWell, now you grasp

Cnst: eine Schraubea screw.

Inst: eine <-> orangene mit einem Schlitz.an <-> orange one with a slit

Cnst: Ja. Inst: Und steckst sie dadurch, also And you put it through, let’s see ….Cnst: von oben

mces in the example from the bielefeld corpus16
MCEs in the example from the Bielefeld corpus

[mce1,ce1| mce1:utter(Inst,“so"), Adv(mce1), ce1:take-turn(Inst), generate(mce1,ce1)];

[mce2,ce2| mce2:utter(Inst,“jetzt"), Adv(mce2), ce2:keep-turn(Inst), generate(mce2,ce2)];

[mce3| mce3:utter(Inst,"nimmst"), Verb(mce3), sem(mce3)= Qx(Q(x’[e| e: grasp(x, x’)]))];

[mce4| mce4:utter(Inst,"Du"), Pro(mce4), sem(mce4)= P.P (you)];

[mce5| mce5:utter(Cnst,"eine"), Det(mce5), sem(mce5)= P’P([y| ]; P’(y); P(y))]

[mce6| mce6:utter(Cnst,"Schraube"), Noun(mce6), sem(mce6)= v([ |screw(v)]];

tag based syntactic interpretation with mces

MCE1 CE1

MCE2 CE2

MCE3

MCE4

mce1:utter(Inst,“so"),ce1:take-turn(Inst), generate(mce1,ce1)];

U1:S

U4:NP

U2:NP

MCE3:”nimmst”:V

U3:NP

MCE4:”Du”:Pro

TAG-based syntactic interpretation with MCEs
incremental interpretation of references
Incremental interpretation of references

M: so we should move the engine at Avon,engine E, to …S: engine E1M: E1S: okayM: engine E1, to Bath, to / or, we could actually move it to Dansville to pick up the boxcar there

two approaches to the semantics of anaphora and accessibility poesio and muskens 1997
Two approaches to the semantics of anaphora and accessibility (Poesio and Muskens, 1997)
  • Accessibility via discourse structure (cfr. Grosz and Sidner / SDRT)
    • Can be extended to MCEs (Poesio and Rieser, in progress)
    • Compatible with view of anaphoric underspecification as homonymy (Poesio, 2001)
    • Problematic for visual situation & dynamics of questions / imperatives
  • Accessibility via resource situations (Poesio, 1993, 1994)
anaphoric expressions and resource situations
Anaphoric expressions and resource situations

the  [u|u:utter(A,”the”), Det(u),sem(u)= K  P  P’ [ x | ]; [ | x = y. K; P(x)]; P’(x) sem(the engine at Avon)  K  P’ [ x | ]; [ | x = y. K; [|engine (y), at(y,Avon))]]; P’(x)

pronouns and resource situations
Pronouns and resource situations

sem(Sie)  K P P’ [ x | ]; [ | x = y. K; [|P(y)]]; P’(x)

resource situations in the discourse situation
Resource situations in the discourse situation

[MapS, u1|

MapS=[a,b,c,d,e, ….. w,s, ……| Avon(a), engine(x),s: at(x,a), Bath(b), ….. ],…..u1: utter(M,”the engine at Avon”),sem(u1) =  K  P’ [ x | ]; [ | x = y. K; [|engine (y),at (y, a))] ]; P’(x)]

incremental reference interpretation
Incremental reference interpretation
  • res-sit(u1)=MapS
  • ( K  P’ [ x | ]; [ | x = y. K; [|engine (y),at (y, a))] ]; P’(x)) (MapS) 
  • sem(u1) =  P’ [ x | ]; [ | x = y. MapS; [|engine (y),at (y,a))] ]; P’(x)
shifting the focus of visual attention
Shifting the focus of (visual) attention

M: while this is happening, take engine E1 to Dansville, pick up the boxcar, and come back to AvonS: okayM: okay then load the boxcar with bananas

the theoretical model mutual visual focus of attention
The theoretical model: mutual (visual) focus of attention
  • Grosz 1977: conversational participants may share a MUTUAL FOCUS OF ATTENTION, which may change in time
    • Participants keep track of where this is
  • Poesio (1992, 1993, 1994a, 1994b):
    • Separate linguistic and visual mutual foci
    • The visual focus is a situation, the MUTUAL SITUATION OF ATTENTION (MSOA)
    • This situation may serve as a resource situation
msoas in the discourse situation
MSOAs in the discourse situation

[MapS, DansvilleS|

MapS=[a,b,c,d,e, ….. w,s, ……| Avon(a), engine(x), s: at(x,a), Bath(b), …..],DansvilleS=[d,x,s | Dansville(d), boxcar(x), s: at(x,d)], …..….MSOA = DansvilleS]

follow the movement
Follow-the-movement
  • Grosz 1977: FOCUS SHIFT RULES
  • FOLLOW THE MOVEMENT (Poesio, 1994)
    • Part of the intended effect of an utterance instructing an agent to move an object from one location to another is to make the terminal location of the movement the new mutual situation of attention.
follow the movement30

Pick up the boxcar

And come back to Avon

Follow the movement

Take engine E1 to Dansville

using msoas as resource situations
Using MSOAs as resource situations

[DansvilleS, u1|

DansvilleS=[d,x,s | Dansville(d), boxcar(x), s: at(x,d)],MSOA = DansvilleS….u1: utter(M,”the boxcar”),sem(u1) =  K  P’ [ x | ]; [ | x = y. K; [|boxcar (y)] ]; P’(x)]

some questions
Some questions
  • Theoretical questions
    • Two models of anaphoric accessibility
    • Formal aspects of the incremental interpretation mechanism
    • Dynamic behaviour of questions and instructions
  • Empirical questions
    • Incremental interpretation
      • Initial interpretation of anaphoric expressions: homonymy or polysemy?
      • Initial resolution: surface only?
    • Follow the movement
      • Effect of motion verbs? Or bridging? (Cfr. Asher and Lascarides, 1998)
empirical developments since 1994
Empirical developments since 1994
  • Psychological side: the Rochester visual world work
  • Corpus side: improved methods for annotating anaphora, now also in dialogue
preliminary answers from psychology
Preliminary answers from psychology
  • Rapid incrementality in reference: Spivey et al, 1995 / Tanenhaus et al, 1995
  • Task constraints are taken into account in selecting the antecedent: Chambers et al 2002
    • Pick up the cube. Now put the cube in a / the can.
    • Also: Altmann and Kamide 1999
  • Brown-Schmidt et al 2004: Referential domains circumscribed to small spatial regions
  • Other evidence for ‘follow the movement’ effects:
    • Stevenson et al, 1993: Bill sent a letter to Sue
    • Matsui 2000: John moved from New Zealand to England. He loves the sheep.
  • But also: challenges to the ‘mutual situation’ account (e.g., (Horton and Keysar, 1995))
further progress on corpus annotation
Further progress on corpus annotation
  • GNOME: detailed coding scheme for anaphora (for texts)
  • VENEX: coding scheme for both text and dialogue (MapTask-style) (in Italian)
  • ARRAU:
    • Further agreement studies
    • Annotation of the TRAINS dialogues
hopes plans for network
Hopes / plans for network
  • Discussions with developers of the other theoretical frameworks represented (Cooper / Ginzburg, Dynamic Tree theory)
  • Other theoretical / computational frameworks for references to the visual situation
    • POTLOG Symposium ?
  • Develop methodology for annotating multimodal data
    • E.g., Donna Byron’s data
  • If possible, design psychological experiments
  • Work also on the ‘other’ side of PTT (grounding and alignment)
the bielefeld toy plane corpus
The Bielefeld Toy Plane Corpus
  • 22 video-filmed, speech recorded and transcribed dialogues
    • two agents, Instructor and Constructor
    • constructing a “Baufix” airplane
    • different sight conditions: total screen, half-screen, face to face
  • 3675 contributions
    • 160 sentence cooperations (4.34 %)
    • in most of them cooperation other-initiated (95%)
common ground beyond assertion
Common ground: beyond assertion

A They have at their disposal enormous assets // and their policy

B //look can I just come in on that// last year

A //YES IN A MINUTE IF YOU MAY AND WHEN I’M FINISHED // then you’ll know

B // yes I’M SO SORRY

(Coulthard 1977)

common ground beyond assertion39
Common ground: beyond assertion

B: Go to Elmhurst, pass the courthouse and go to Elmhurst and then to Elmhurst, uh north.A: mm hum.B: Towards Riverton, till you come to that Avila HallA: Oh yesB: Dju know where that//is?A: //uh huhA: Oh surelyB: Avilla Hall on the corner of Bor//donA: //uh huhB: Well there, on Bordon you turn back to town, left.

(George Psathas, "Direction-giving in Interaction," in Boden and Zimmerman, ed.)

locutionary acts in the common ground
Locutionary acts in the common ground

“The fact that a speaker is speaking, saying the words he is saying in the way he is saying them, is a fact that is usually accessible to everyone present. Such observed facts can be expected to change the presumed common background knowledge of the speaker and his audience in the same way that any obviously observable change in the physical surroundings of the conversation will change the presumed common knowledge.” (Stalnaker, Assertion, p. 323)

the time order of sentence processing
The time-order of sentence processing
  • GARDEN-PATH phenomena shows that parsing is INCREMENTAL (Bever, 1974; Frazier, 1987)
  • Marslen-Wilson 1973, 1975: semantic information ALSO accessed immediately
  • Swinney, 1979: lexical access incremental
  • Just and Carpenter,1980: IMMEDIACY HYPOTHESIS (“Every word encountered should be processed to the deepest level possible before the eye moves on to the next word”)
  • Eye-tracking work (Tanenhaus et al, 1995, tomorrow): really fine-grained incrementality
clarification questions ginzburg and cooper purver and ginzburg
Clarification questions (Ginzburg and Cooper, Purver and Ginzburg)

A: Did Bo leave?B: BO? A: Bo Smith.B: Yes, half an hour ago.

Matthew: It wasn’t all that bad. At least the pool was clean.Lara: MR POOL?Matthew: The pool.Lara: Oh. <laugh>

(“What is the intended content of your utterance ‘Bo’?”)

(“Did you utter the words ‘Mr. Pool’?”)

semantic interpretation and compositionality

U3:()

U1:

U2: 

Semantic interpretation and compositionality

BINARY SEMANTIC COMPOSITION

intentions and obligations
Intentions and obligations

OBLIGATIONS: [o | o:OblCnst ([|address(Cnst, ce1)])]

INTENTIONS: [i | i:IntInst&Cnst ([|join(Cnst, wing1,fuselage1)])]

INTENTIONAL STRUCTURE: Grosz&Sidner-like sp(i1) = i2 dom(i1) = i2

grounding
Grounding
  • As in proposals such as Clark and Schaefer (1989) and Traum (1994), establishment of common ground (‘G’) modeled in terms of CONTRIBUTIONS, or DISCOURSE UNITS, that may be ACKNOWLEDGED or REPAIRED
discourse units and grounding acts

DU1 …. DU17

MCE1 CE1

MCE2 CE2

MCE3 …

ACK(DU17)

mce1:utter(Inst,“so"),ce1:take-turn(Inst), generate(mce1,ce1)];

DU17 =

MCE4:”Du”:Pro

U1:S

CONT(DU17)

U2:NP

MCE3:”nimmst”:V

U3:NP

REPAIR(DU17)

Discourse Units and Grounding Acts
an iam analysis of the btpc example
An IAM-analysis of the BTPC example
  • What leads Cnst to produce “eine Schraube”?
  • What is a situation model in this domain?
    • “the key dimensions encoded in situation models are SPACE, TIME, CAUSALITY, INTENTIONALITY, and REFERENCE to the MAIN INDIVIDUALS … “ (p. 7)
preliminary conclusions
Preliminary conclusions
  • PTT provides the technical tools to formalize a crucial feature of sentence cooperations: coordination at the micro conversational event level
  • Mind-reading always difficult, but Tuomela’s theory of we-intention goes some way towards formalizing one of the possible motivations for completions, in terms of “help”
  • A preliminary investigation of the alignment route also possible