1 / 113

Multi-Perspective Question Answering

Multi-Perspective Question Answering. ARDA NRRC Summer 2002 Workshop. Janyce Wiebe Eric Breck Chris Buckley Claire Cardie Paul Davis. Bruce Fraser Diane Litman David Pierce Ellen Riloff Theresa Wilson. Participants. Part I: Overview.

elevi
Download Presentation

Multi-Perspective Question Answering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multi-Perspective Question Answering ARDA NRRC Summer 2002 Workshop

  2. Janyce Wiebe Eric Breck Chris Buckley Claire Cardie Paul Davis Bruce Fraser Diane Litman David Pierce Ellen Riloff Theresa Wilson Participants

  3. Part I: Overview

  4. Finding and organizing opinions in the world press and other text Problem

  5. Our Work will Support • Finding a range of opinions expressed on a particular topic, event, issue • Clustering opinions and their sources • Attitude (positive, negative, uncertain) • Basis for opinion (supporting beliefs, experiences) • Expressive style (sarcastic, vehement, neutral) • Building perspective profiles of individuals and groups over many documents and topics

  6. Task: Conceptualization • Perspective in language • Implications for higher-level tasks

  7. Manual annotation scheme for linguistic expressions of opinions “It is heresy,” said Cao. “The `Shouters’ claim they are biggerthan Jesus.” Task: Annotation (writer,Cao) (writer,Cao,Shouters) (writer,Cao) (writer,Cao)

  8. ATTITUDE value: neg strength: medium <writer,many countries>: <writer>: <report> subjectivity index: 1.0 (1/1) subjectivity index: 0.4 (4/10) ATTITUDE value: neg strength: low ATTITUDE value: neg strength: high <US> Task: Summary Representation of Opinions • Built upon lower-level annotations

  9. Task: Automate Manual Annotations • Machine learning • Identification of opinionated phrases, sources of opinions, …

  10. Task: Organizing Perspective Segments • Unsupervised clustering • Text features + features from the annotation scheme + higher-level features

  11. Evaluation • Exploratory manual clustering • Evaluation of annotation agreement • Evaluation of automatic annotations • Evaluation of automatic clustering

  12. Part II: Conceptualization

  13. Introduction • Perspective – Framework/filter through which we see an event • Language give insight into perspective of writer/ speaker and the people they describe • Our focus: subjectivity (reflections of private states) as conveyed through language • Goal: Draw from and elaborate on the Annotation Guidelines to expand our knowledge of how subjectivity is conveyed via linguistic expressions.

  14. Influences on Perspective • Identity (President, Assistant Secretary of State) • Attitudes held by a person: • Micro (towards specific individuals, events) • Macro (belief systems embraced, culture values) • Genre of the product (editorial, report, white paper) • Context of the writing: • Temporal (writing now about 1960) • Spatial (writing located in war zone) • Events (writing now rather than before 9-11)

  15. Lexical Clues to Subjectivity Attitude Expression <Source, Attitude, Object> • Explicit Attitudes – Direct realization of private states • Implicit Attitudes – Expressive subjective elements • Other linguistic clues

  16. Attitude Expression • The Source (writer, speaker, another person an institution, a document • The Object (entities, actions, propositions, topic, context) • Attitudes – Different typologies

  17. Attitude Typologies • Negative (reject); Positive (enjoy); Other (surprise) • Opinion (criticize); Emotions (fear); Speculation (probably) • Other (must, poor, better)

  18. Attitude Types • Content Evidential (allegedly, think) Deontic (require, should) Assessment (like, excellent) • Process Discourse Markers (so, but) Operators (very, not)

  19. Types of AE Combinations • Reinforced Attitude (A+) – Two AEs of the same type (I like her and I admire her as well) • Weakened Attitude (A-) – Two AEs, A and ~A (I like her. I can’t stand her politics) • Entailed Attitude (=>A)- AE A entails AE B (He terrorizes people => people fear) • Modified Attitude (A(B)) – AE A modifies AE B (He cruelly criticized the people)

  20. Subjectivity: Relation to Ideology • Ideology – Beliefs and opinions held • Approximation of ideology by grouping AEs for a given Source • Grouping Sources with similar ideologies • Inferring ideology from limited AEs • Change of an ideology over time • Prediction of position based on ideology

  21. Solution Architecture Annotation Architecture AnnotationTool Learning Architecture LearningAlgorithms Trained Taggers Application Architecture DocumentRetrieval PerspectiveTagging DocumentClustering Question Other Taggers

  22. Part III: Annotation

  23. Annotation • Find opinions, evaluations, emotions, speculations (private states)expressed in language Private state: state that is not open to objective observation or verification. Quirk, Greenbaum, Leech, Svartvik (1985). A Comprehensive Grammar of the English Language.

  24. Two Ways of Expressing Private States • Explicit mentions of private states and speech events • The United States fears a spill-over from the anti-terrorist campaign • Expressive subjective elements • The part of the US human rights report about China is full of absurdities and fabrications.

  25. Two Ways of Expressing Private States • Explicit mentions of private states and speech events: “Ons” • The United States fears a spill-over from the anti-terrorist campaign • Expressive subjective elements • The part of the US human rights report about China is full of absurdities and fabrications.

  26. (writer, Xirao-Nima, US) (writer, Xirao-Nima) (writer, Xirao-Nima) (writer, Xirao-Nima) “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer) (writer) “The report is full of absurdities,’’ he continued. Nested Sources

  27. OnlyFactive=no (writer, Xirao-Nima, US) (writer, Xirao-Nima) (writer) OnlyFactive=yes OnlyFactive OnlyFactive=yes “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities.

  28. OnlyFactive • An example of all OnlyFactive=yes The government, it added, has amended Pakistan Citizenship Act 10 of 1951 to enable women of Pakistani descent to claim Pakistani nationality for their children born to foreign husbands.

  29. “It is heresy,” said Cao. “The `Shouters’ claim they are bigger than Jesus.” (writer,Cao) (writer,Cao,Shouters) (writer,Cao) (writer,Cao) Example

  30. (writer,FM) (writer,FM,FM) (writer,FM) (writer,FM,FM,SD) (writer,FM) (writer,FM) Example The Foreign Ministry said Thursday that it was “surprised, to put it mildly” by the U.S. State Department’s criticism of Russia’s human rights record and objected in particular to the “odious” section on Chechnya.

  31. Other attributes • Strengths of various components • Types of private states • Objects

  32. Documents and Annotators • Over 90 annotated documents • 100%: spans of sources, subjective elements, ons, with type and OnlyFactive attributes. • 39%: + sources, strengths, agent objects • Participants + part-time annotators

  33. Pilot Agreement Evaluation:ons and subjective elements • Agreement measured by percentage of A’s constituents matching B’s, and vice versa

  34. Overlapping Spans: Ons Bush has adopted the most pro-Taiwan posture of any president... Bush has adopted the most pro-Taiwan posture of any president...

  35. Overlapping Spans: Expressive Subjective Elements …some of Mr. Chavez’s more alarming faults… …some of Mr. Chavez’s more alarming faults…

  36. Untrained Annotators • 2 groups of 3 untrained annotators (the participants) each annotated 4 documents without any communication

  37. Spans, Untrained Annotators

  38. Overlapping Spans: Expressive Subjective Elements …some of Mr. Chavez’s more alarming faults… …some of Mr. Chavez’s more alarming faults…

  39. C: 74 P: 196 D: 75 Expressive Subjectivity Agreement Di: 76 T: 153 E: 29; 97%

  40. Spans, All Annotators

  41. Pilot Agreement Evaluation:OnlyFactive • For the onsidentified by both annotators: Measured agreement for Onlyfactive • Cohen’s Kappa • Krippendorf: 0.67+ Tentative conclusions 0.80+ Definite conclusions

  42. OnlyFactive Kappa Values

  43. Future work • Continue annotations by trained annotators • Continue to explore annotations • Train new annotators, starting with settled scheme • Even more detail is possible

  44. Part IV: Summary Representations of Opinions

  45. Example The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  46. Sentence 1 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  47. <writer>:onlyfactive<writer>:expr-subj (medium) Sentence 1: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  48. Sentence 2 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  49. Sentence 2: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>: neg-attitude (medium) <report> <writer>: neg-attitude (medium) <writer>: neg-attitude (medium)

More Related