1 / 27

Dialogue Annotation at SUNY

Dialogue Annotation at SUNY. Hilda Hardy Amiti é s Annotation Workshop LIMSI-CNRS, France July 8-10, 2002. What did we annotate?. Transcriptions of dialogues from 3 folders in the englishPlainTextDialogues.zip archive (dated March 28, 2002): 22mar02, 20 of 60 files (Uzo Enyinna)

katima
Download Presentation

Dialogue Annotation at SUNY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dialogue Annotation at SUNY Hilda Hardy Amitiés Annotation Workshop LIMSI-CNRS, France July 8-10, 2002

  2. What did we annotate? Transcriptions of dialogues from 3 folders in the englishPlainTextDialogues.zip archive (dated March 28, 2002): • 22mar02, 20 of 60 files (Uzo Enyinna) • 22mar02, 21 of 60 files (Hilda Hardy) • 26feb02, 30 of 64 files (Tom Palen) • 26mar02, 36 of 36 files (Hilda Hardy)

  3. What approach did we use? We annotated the Semantic and Functional layers of the dialogues (not the Stylistic layer), using XDMLTool version 2.0.

  4. Why annotate Semantic and Functional Aspects of Real Call-Center Dialogues? • Get ideas and data for designing a better, more human-like Dialogue Manager • Find out structures of dialogues • Basic patterns of Dialogue Acts • Basic flow of the dialogue • What is essential in accomplishing the task? • What is parenthetical? • Find out how information is exchanged • How is information broken down? • How is information ordered in a dialogue? • Discover relationships between functional and semantic layers

  5. Why annotate Semantic and Functional Aspects? Cont’d • What are the most common questions customers ask agents? • What facts do agents need to learn from customers? What facts do they give customers? We need supplementary information from the call centers to handle these aspects. • Improve language generation • How do agents converse with customers? Look at phrasing, length of turns • Decide what can be automated • Other: a well-annotated corpus is a valuable contribution to the research community

  6. Semantic Annotation: Customer AccessFrames Customer transactions: every dialogue has a purpose: at least one topic or task that the customer needs to have answered or satisfied. Task Information Fees LostCard Insurance ChangeAddress Statement CloseAccount AcctBalance MakePayment CancelDebit

  7. Semantic Annotation: Agent AccessFrames Agent transactions: tasks initiated by the agent in the process of satisfying the customer’s needs GreetCaller “Good afternoon, . . . How can I help?” Information required for security checks: name, old post code, date of birth, usual method of payment, etc. VerifyId Closure “Is there anything else? . . . Goodbye”

  8. Semantic Annotation: Attributes, Values and Modifiers Most frames are associated with details that may be organized into key-value pairs. Examples: AccessFrame Attribute Value Modifier Name Paul Agent PostCode AB1 2CD New HouseNumber 10 New GreetCaller ChangeAddress

  9. Semantic Annotation: General Shape of a Call Center Dialogue A typical call center dialogue progresses with this general order of frames: GreetCaller VerifyId ChangeAddress or or MakePayment Info-Insurance Closure

  10. Semantic Annotation: example GreetCaller ChangeAddress VerifyId A: Good afternoon customer services Paul speakingC: Hello I want to inform you of my change of address pleaseA: Certainly what name is it please?C: Name? It's Mrs Pauline SmithA+C: Right Mrs Hunter I'm just keying in the account number for your bear with me a momentUh hmmA: (pause)Could you confirm the, your previous postcode and telephone number please?C: Yeah the previous one was AB1 2CD01111 111111A: Thank you, you're speaking to Paul, good afternoonC: Uh hmmA: Right so it's Paul and PaulineC: (laughs)A: The new address, is it a house number Mrs Smith?C: Yeah it's number 3A: And the postcode there is?C: W for Whiskey X for X-ray two ohA: Yes GreetCaller ChangeAddress

  11. Semantic Annotation: example, cont’d ChangeAddress C: Three Y for Yankee Z for ZuluA: And is that The Oaks?C: That's rightA: The Hall?C: That's itA: Do you have a new telephone number?C: Yes I do it's erm 01111A: YesC: 111A: YesC: 111A+C: And this is your Principles card isn't it?YeahA+C: Have you got any other shop cards that you need to changeNo, no I haven't noA: FineA+C: Is there anything else I can do for you?OkayC: That's all thank you very muchA: Thanks for your callC: Thank you byeA: Bye(END OF CALL) Closure

  12. Semantic Annotation with XDMLTool Semantic categories

  13. Semantic Annotation with XDMLTool • Semantic annotation is domain-dependent • But the categories AccessFrame, Attribute, Value, and Modifier were designed to be domain-independent. • For a new domain, use new lists of tags. • Helpful to begin with a set of labels developed during preliminary annotation • Necessary to allow the annotator to add to the lists as new information appears in the data

  14. Semantic Annotation: Suggestions • Compare annotators’ lists and merge labels that are clearly the same • ChangeAddress and UpdateAddress • VerifyId and VerifyCallerID • Card and NameofCard • Discuss how to handle cases that are not so obvious: What level of detail is best? • Address = StreetAddr, Area/Town, City, County? • Try to finalize lists as far as possible, leaving room for new labels

  15. Functional Annotation with XDMLTool Functional categories

  16. Functional Annotation: General Approach • Annotate dialogue act functions (dialogue-related speech acts, or DAs) • Use DAMSL (Dialog Act Markup in Several Layers), tailored to reflect call center dialogue features • Both the categories and the lists are domain-independent • Minor modifications to DAMSL are necessary • Annotation conventions must be discussed

  17. DAMSL Amitiés/XDML (proposed) Communicative Status (Features) Self-talk used occasionally Uninterpretable not used (seems better suited to transcription) Abandoned used occasionally Non-verbal (keyboard noise, music, dog barking) Third-party-talk (side conversations)

  18. DAMSL Amitiés/XDML (proposed) Information-Level Task annotated by default Task-management not used (rare; agent reacts to the customer) Communication-mgt used as defined Other-level Out-of-topic (jokes, non-sequiturs, small talk, and any comments or remarks that have no direct bearing on accomplishing the task)

  19. DAMSL Amitiés/XDML (proposed) Forward Looking Function Statement Assert used for simple assertions, such as answers Explanation(agent explains policies, customer gives reasons) Recap (agent summarizes, wraps up, or recapitulates the task) Reassert used as defined Other-statement not used

  20. DAMSL Amitiés/XDML (proposed) Influence-on-listener Action-directive used as defined Open-option used as defined (when a course of action is suggested with no obligation on the listener) Info-request used only to request information, not to elicit a yes/no response Confirmation-request (needs a yes/no response)

  21. DAMSL Amitiés/XDML (proposed) Influence-on-speaker Offer used as defined Commit used as defined Conventional Opening used as defined Closing used as defined Explicit-performative Exclamation Expression (thank you, okay, Other-forward-function no problem, smashing, sorry)

  22. DAMSL Amitiés/XDML (proposed) Backward Looking Function Agreement Accept Accept-part Maybe used as defined Hold Reject Reject-part I_don’t_know

  23. DAMSL Amitiés/XDML (proposed) Understanding Signal-non-understanding Non-understanding Signal-understanding Acknowledge used as defined* Repeat-rephrase used as defined Completion used as defined Correct-misspeaking Correction Answer used as defined *Same definition as “Backchannel”. NOTE: we recommend removing “Confirm” and “Disconfirm” (same as Accept and Reject)

  24. Annotation: Other Issues • When should an annotator split (segment) a turn? • Semantic data must be annotated separately: C: Post code AB1 2CD, phone 01111 111111 • One turn may have different functions: C: Hi there erm I’ve lost my card and also I’ve changed address • An overlapping turn has different speakers: A+C: It’s Top Shop account card Top Shop thanks

  25. Annotation: Other Issues • Leave the turn intact when it begins with an acknowledgment. Info-level is “Task” and not “Communication-mgt.” A+C: Top Shop thanks A: Okay what’s your surname? A: Ok and your address, the old one that is?

  26. Automatic tag selections Several of the DAMSL tags are contained in others or require other selections to be made. For example, "answers by definition will always be asserts." (Draft of DAMSL, p. 23). For added convenience, XDMLTool v. 2.0 makes the following additional selections by default. If exceptions arise, the user may change the automatic selections manually. When annotator selects... ... XDMLTool selects... any Backward Communicative Function Previous utterance number as "Response-to" value Conventional->Opening Info-level->Communication-mgt Conventional->Closing Info-level->Communication-mgt Understanding->Acknowledge Info-level->Communication-mgt Understanding->Backchannel Info-level->Communication-mgt Understanding->Non-understanding Info-level->Communication-mgt Answer Statement->Assert Agreement->Accept Answer, Statement->Assert Agreement->Accept-part Answer, Statement->Assert Agreement->Reject Answer, Statement->Assert Agreement->Reject-part Answer, Statement->Assert

More Related