Semantic Annotation Evaluation and Utility

1. Semantic Annotation Evaluation and Utility Bonnie Dorr Saif Mohammad David Yarowsky Keith Hall

2. Road Map • Project Organization • Semantic Annotation and Utility Evaluation Workshop • Focus Area: Informal Input • Belief/Opinion/Confidence (modality) • Dialog Acts • Complex Coreference (e.g., events) • Temporal relations • Interoperability • Current and Future Work

3. Project Organization CMU (Mitamura, Levin, Nyberg) Coreference Entity relations Committed Belief BBN (Ramshaw, Habash) Temporal Annotation Coreference (complex) Evaluation Bonnie Dorr David Yarowsky Keith HallSaif Mohammad UMBC (Nirenburg, McShane) Modality: polarity, epistemic, belief, deontic, volitive, potential, permissive, evaluative Columbia (Rambow, Passonneau) Dialogic Content Committed Belief Affiliated Efforts Ed Hovy Martha Palmer George Wilson (Mitre)

4. Semantic Annotation & Utility Evaluation Meeting: Feb 14th Site presentations included an overview of the phenomena covered and utility-motivating examples, extracted from the target corpus. Collective assessment of what additional capabilities could be achieved if a machine could achieve near human-performance on annotation of these meaning layers relative to applications operating on text without such meaning layer analysis. Compatibility, Interoperability, integration into larger KB environment. How can we automate these processes?

5. Attendees • Kathy Baker (DoD) • Mona Diab (Columbia) • Bonnie Dorr (UMD) • Tim Finin (JHU/APL) • Nizar Habash (Columbia) • Keith Hall (JHU) • Eduard Hovy (USC/ISI) • Lori Levin (CMU) • James Mayfield (JHU/APL) • Teruko Mitamura (CMU) • Saif Mohammad (UMD) • Smaranda Muresan (UMD) • Sergei Nirenburg (UMBC) • Eric Nyberg (CMU) • Doug Oard (UMD) • Boyan Onyshkevych (DoD) • Martha Palmer (Colorado) • Rebecca Passonneau (Columbia) • Owen Rambow (Columbia) • Lance Ramshaw (BBN) • Clare Voss (ARL) • Ralph Weischedel (BBN) • George Wilson (Mitre) • David Yarowsky (JHU)

6. Analysis of Informal Input: Unifies Majority of Annotation Themes • Four relevant representational Layers: • Belief/Opinion/Confidence (modality) • Dialog Acts • Coreference (entities and events) • Temporal relations • Many relevant applications: • KB population • Social Network Analysis • Sentiment analysis • Deception detection • Text mining • Question answering • Information retrieval • Summarization • Analysis of informal input is dynamic: a first analysis may be refined when subsequent informal input contributions are processed

7. Representational Layer 1: Committed Belief • Committed belief: Speaker indicates in this utterance that Speaker believes the proposition • I know Afghanistan and Pakistan have provided the richest opportunity for Al Qaeda to take root. • Non-committed belief: Speaker identifies the proposition as something which Speaker could believe, but Speaker happens not to have a strong belief in the proposition • Afghanistan and Pakistan may have provided the richest opportunity for Al Qaeda to take root. • No asserted belief: for Speaker, the proposition is not of type in which Speaker is expressing a belief, or could express a belief. Usually, this is because the proposition does not have a truth value in this world. • Did Afghanistan and Pakistan provide the richest opportunity for Al Qaeda to take root?

8. Committed Belief is not Factivity CB = committed belief, NA = No asserted belief • Committed-belief annotation and factivity annotation are complementary • NA cases may lead to detection of current and future threats, sometimes conditional. Multiple modalities (opinion detection): • Potential: “Smith might be assassinated — if he is in power.” • Obligative: “Smith should be assassinated.”

9. Committed Belief is not Tense CB = committed belief, NA = No asserted belief • Special feature to indicate future tense on CB (committed belief) and NCB (non-committed belief)

10. Why Is RecognizingCommitted Belief Important? • Committed-Belief Annotation Distinguishes • Propositions that are asserted as true (CB) • Propositions that are asserted but speculative (NCB) • Propositions that are not asserted at all (NA) • Important whenever we need to identify facts • IR Query: show documents discussing instances of peasants being robbed of their land • Document found 1: The people robbing Iraqi peasants of their land should be punished RELEVANT: YES • Document found 2: Robbing Iraqi peasants of their land would be bad. RELEVANT: NO • QA: Did the humanitarian crisis in Iraq end? • Text found 1: He arrived on Tuesday, bringing an end to the humanitarian crisis in Iraq. ANS: YES. • Text found 2: He arrived on Tuesday, calling for an end to the humanitarian crisis in Iraq. ANS: I DON’T KNOW

11. Representational Layer 2: Dialog Acts • INFORM • REQUEST-INFORMATION • REQUEST-ACTION • COMMIT • ACCEPT • REJECT • BACKCHANNEL • PERFORM • CONVENTIONAL

12. Why is dialog analysis important? • Understanding the outcome of an interaction • What is the outcome? • Who prevailed? • Why (status of interactants, priority of communicative action)? • Application of a common architecture to automatic analysis of interaction in email, blogs, phone conversations, . . . • Social Network Analysis: Is the speaker/sender in an inferior position to the hearer/receiver? • How can we know? (e.g., REJECT a REQUEST)

13. Representational Layer 3: Complex Coreference (e.g., events) Annotate events beyond ACE coreference definition • ACE does not identify Events as coreferents when one mention refers only to a part of the other • In ACE, the plural event mention is not coreferent with mentions of the component individual events. • ACE does not annotate: “Three people have been convicted…Smith and Jones were found guilty of selling guns…” “The gunman shot Smith and his son. ..The attackagainst Smith.”

14. Related Events (and sub-events) • Events that happened “Britain bombed Iraq last night.” • Events which did not happen “Hall did not speak about the bombings.” • Planned events planned, expected to happen, agree to do… “Hall planned to meet with Saddam.” • Sub-Event Examples: • “drug war” (contains subevents: attacks, crackdowns, bullying…) • “attacks” (contains subevents: deaths, kidnappings, assassination, bombed…)

15. Why is complex coreference resolution important? • Complex Question Answering: • Event questions: Describe the drug war events in Latin America. • List questions: List the events related to attacks in the drug war. • Relationship questions: Who is attacking who?

16. Representational Layer 4: Temporal Relations Baghdad 11/28 -- Senator Hall arrived in Baghdad yesterday. He told reporters that he “ will not be visiting Tehran” before he left Washington. He will return next Monday. TimeUnitType Relation Parent 11/28 Specific.Date After arrived arrived Past.Event Before <writer> yesterday Past.Date Concurrent arrived told Past.Say Before arrived visiting Neg.Future.Event After told left Past.Event After told return Future.Event After <writer> Monday Specific.Date Concurrent return

17. Temporal Relation Parse <writer> arrived return told yesterday 11/28 Monday left (not) visiting TIME

18. Temporal Relation Analysis:Inter-annotator Agreement

19. Why is Temporal Analysis Important? • Constructing activity schedules from text • Question answering (temporal):did/does/will X happen before/after/same_time_with Y?where X,Y are events, states, dates or time ranges.

20. Interoperability: Data • Common data model • Multiple implementations • based on the same underlying schema(formal object model) • meet different goals / requirements • Implementation Criteria: • Support effective run-time annotation • Support effective user interface, query/update • Support on-the-fly schema extension

21. Example: UMBC Modality Annotations 21

22. Ongoing and Future work • Move to new genre—informal input. • Establish compatibility across levels. • Continue examining intra-site and cross-site annotation agreement rates • Initial assessment of computational feasibility of machine learning approaches—“our annotations are supposed to be fodder for ML approaches.” • Implementation of framework for superimposing semantic “layers” on existing objects (e.g., on top of ACE types). • Move to multiple languages.