1 / 21

You talking to me? A Corpus and Algorithm for Conversation Disentanglement

You talking to me? A Corpus and Algorithm for Conversation Disentanglement. Micha Elsner and Eugene Charniak Brown University ACL 2008. Research Objective. Conversation Disentanglement. Research Objective. (Chanel) Felicia: google works :)

tiara
Download Presentation

You talking to me? A Corpus and Algorithm for Conversation Disentanglement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. You talking to me? A Corpus and Algorithm for Conversation Disentanglement Micha Elsner and Eugene Charniak Brown University ACL 2008

  2. Research Objective • Conversation Disentanglement

  3. Research Objective (Chanel) Felicia: google works :) (Gale) Arlie: you guys have never worked in a factory before have you (Gale) Arlie: there’s some real unethical stuff that goes on (Regine) hands Chanel a trophy (Arlie) Gale, of course ... thats how they make money (Gale) and people lose limbs or get killed (Felicia) excellent

  4. Research Objective (Chanel) Felicia: google works :) (Gale) Arlie: you guys have never worked in a factory before have you (Gale) Arlie: there’s some real unethical stuff that goes on (Regine) hands Chanel a trophy (Arlie) Gale, of course ... thats how they make money (Gale) and people lose limbs or get killed (Felicia) excellent

  5. Research Objective • Conversation disentanglement • New corpus • New annotator-agreement metrics

  6. Application • Public chat • QA

  7. Outline • Research Objective • Corpus • Annotator-Agreement Metrics • Disentanglement Method • Experiment

  8. New Conversation Corpus • IRC (Internet Relay Chat) • Linux topic • Training: 706 utterances (2:06 hr) • Testing: 800 utterances (1:39 hr) • 7 university student annotators

  9. Outline • Research Objective • Corpus • Annotator-Agreement Metrics • Disentanglement Method • Experiment

  10. New Annotator-Agreement Metrics • 1-to-1 Accuracy • Pair up conversations from 2 annotators • Maximize overlap percentage • Local Agreement • Is each of previous k utterances from the same conversation as the current utterance? • Determine annotator agreement

  11. Annotator Agreement Results for Test Corpus

  12. Outline • Research Objective • Corpus • Annotator-Agreement Metrics • Disentanglement Method • Experiment

  13. Automatic Conversation Identification • 2 Steps • Utterance pair judgment • Cluster

  14. Utterance Pair Classification • Maximum Entropy Classifier

  15. Features and Inside Test Results • Outside Test: Acc 68.2, Prec 53.3, Rec 71.3, F 60

  16. Utterance Pair Time Difference

  17. Cluster Method • Window size n • Choose most similar preceding utterance within window or create a new conversation

  18. Outline • Research Objective • Corpus • Annotator-Agreement Metrics • Disentanglement Method • Experiment

  19. Automatic Conversation Identification Baselines • All different • All same • Blocks of k • k consecutive utterances • Pause of k • Within k seconds • Speaker • 1 conversation/speaker

  20. Experimental Results

  21. Resource • http://cs.brown.edu/people/melsner

More Related