multi perspective question answering
Download
Skip this Video
Download Presentation
Multi-Perspective Question Answering

Loading in 2 Seconds...

play fullscreen
1 / 65

Multi-Perspective Question Answering - PowerPoint PPT Presentation


  • 271 Views
  • Uploaded on

Multi-Perspective Question Answering ARDA NRRC Summer 2002 Workshop Janyce Wiebe Eric Breck Chris Buckley Claire Cardie Paul Davis Bruce Fraser Diane Litman David Pierce Ellen Riloff Theresa Wilson Participants Finding and organizing opinions in the world press and other text Problem

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Multi-Perspective Question Answering' - Ava


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
multi perspective question answering

Multi-Perspective Question Answering

ARDA NRRC Summer 2002 Workshop

participants
Janyce Wiebe

Eric Breck

Chris Buckley

Claire Cardie

Paul Davis

Bruce Fraser

Diane Litman

David Pierce

Ellen Riloff

Theresa Wilson

Participants
our work will support
Our Work will Support
  • Finding a range of opinions expressed on a particular topic, event, issue
  • Clustering opinions and their sources
    • Attitude (positive, negative, uncertain)
    • Basis for opinion (supporting beliefs, experiences)
    • Expressive style (sarcastic, vehement, neutral)
  • Building perspective profiles of individuals and groups over many documents and topics
task annotation
Manual annotation scheme for linguistic expressions of opinions

“It is heresy,” said Cao. “The `Shouters’ claim

they are biggerthan Jesus.”

Task: Annotation

(writer,Cao)

(writer,Cao,Shouters)

(writer,Cao)

(writer,Cao)

task annotation6
(writer,FM)

(writer,FM,FM)

(writer,FM)

(writer,FM,FM,SD)

(writer,FM)

(writer,FM)

Task: Annotation

The Foreign Ministry said Thursday that it was “surprised, to put it mildly”

by the U.S. State Department’s criticism of Russia’s human rights

record and objected in particular to the “odious” section on Chechnya.

task conceptualization
Task: Conceptualization
  • Various ways perspective is manifested in language
  • Implications for higher-level tasks
task automate manual annotations
Task: Automate Manual Annotations
  • Machine learning
  • Identification of opinionated phrases, sources of opinions, …
task organizing perspective segments
Task: Organizing Perspective Segments
  • Unsupervised clustering
  • Text features + features from the annotation scheme + higher-level features
solution architecture
Solution Architecture

Annotation Architecture

AnnotationTool

Learning Architecture

LearningAlgorithms

Trained Taggers

Application Architecture

DocumentRetrieval

PerspectiveTagging

SegmentClustering

Question

Other Taggers

evaluation
Evaluation
  • Exploratory manual clustering
  • Evaluation of automatic annotations against manual annotations
  • End-user evaluation of how well the system groups text segments into clusters of similar opinions about a given topic
  • Development of other end-user evaluation tasks
example
Example

The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media.

Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries.

Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

example13
:fact

neg-attitude the report

:fact

Many countries in Asia, Europe, Africa, and Latin America neg-attitude the content

of the US Human Rights Report neg-attitude it

:fact

a report on

human rights in the United States in 2001 neg-attitude there

:fact

the report:fact

:fact

:subjective

:fact

:fact

the report:fact

:subjective

Example

The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media.

Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries.

Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

example14
neg-attitude

:

:

:


subjectivity index: 4/10=40%

subjectivity index: 2/2=100%

subjectivity index: 1/3=33%

(medium)

expressive style: medium

expressive style: extreme

expressive style: medium

neg-attitude

(strong)

(medium)

Example

neg-attitude

support the following
Support the following…
  • Describe the collective perspective w.r.t. issue/object presented in an individual article, across a set of articles, …
  • Describe the perspective of a particular writer/individual/government/news service w.r.t. issue/object in an individual article, across a set of articles, …
  • Create a perspective profile for agents, groups, news sources, etc.
outline
Outline
  • Annotation: Wiebe & Wilson
  • Conceptualization: Davis
  • Architecture: Pierce
  • End-user evaluation: Buckley
annotation
Annotation
  • Find opinions, evaluations, emotions, speculations (private states) expressed in language
annotation18
Annotation
  • Explicit mentions of private states and speech events
    • The United States fears a spill-over from the anti-terrorist campaign
  • Expressive subjective elements
    • The part of the US human rights report about China is full of absurdities and fabrications.
annotation19
Nested sources

(writer,Xirao-Nima,US)

(writer,Xirao-Nima)

“The report is full of absurdities,’’ he continued.

Annotation

“The US fears a spill-over’’, said Xirao-Nima, a professor

of foreign affairs at the central university for nationalities.

annotation20
Annotation
  • Whether opinions or other private states are expressed in speech
  • Type of private state (negative evaluation, positive evaluation, …)
  • Object of positive or negative evaluation
  • Strengths of expressive elements and private states
example21
“It is heresy,” said Cao. “The `Shouters’ claim

they are bigger than Jesus.”

(writer,Cao)

(writer,Cao,Shouters)

(writer,Cao)

(writer,Cao)

Example
example22
(writer,FM)

(writer,FM,FM)

(writer,FM)

(writer,FM,FM,SD)

(writer,FM)

(writer,FM)

Example

The Foreign Ministry said Thursday that it was “surprised, to put it mildly”

by the U.S. State Department’s criticism of Russia’s human rights

record and objected in particular to the “odious” section on Chechnya.

accomplishments
Accomplishments
  • Fairly mature annotation scheme and instructions
  • Representation supporting manual annotation using GATE (Sheffield)
  • Annotation corpus
  • Significant training of 3 annotators
  • Participants understand the annotation scheme
conceptualization
Conceptualization
  • Ideology, emotions, and opinions are reflected in language
  • Language gives us a means to track and assess perspective
  • Goal: create document to support workshop annotation and experiments, and to extend to future applications
conceptualization part i theoretical background
Conceptualization Part I:Theoretical Background
  • Types of perspective: attitudes (subjectivity), spatial, temporal, sociological, etc.
  • Focuses on subjectivity expressed linguistically (e.g., opinions: criticized an unfair election, emotions: applaudedthe election, speculations: probably will be elected)
conceptualization subjectivity theoretical background continued
Conceptualization: SubjectivityTheoretical Background (continued)
  • Sources have Attitudes about Objects: (writer, criticizes, election)
  • An ontology of attitudes leading to different types of private states (distinctions can range from identification, to positive and negative, to more fine-grained: reliability, source, assessment, necessity, etc.)

This theoretical background informs the annotation strategy, experiments, and extensions

conceptualization part ii looking to higher levels larger segments
Conceptualization Part II:Looking to higher levels: larger segments
  • Subjectivity beyond the immediate occurrence of the segment:
    • sentence and paragraph level
    • document level
    • discourse and topic level
conceptualization part iii looking forward applications
Conceptualization Part III:Looking forward: applications
  • Track perspective over time (identify changes)
  • Identify ideology (subjective expressions taken as a unit may approximate ideology)
  • Cluster agents with similar ideologies

(similar expressions of opinions may help group those on the same side)

  • Infer ideology from limited expressions of perspective (some subjectivity for a source may suggest opinions on other topics)
architecture overview
Architecture Overview
  • Solution architecture includes:
    • Application Architecture
      • supports high-level QA task
    • Annotation Architecture
      • supports document annotation
    • Learning Architecture
      • supports development of low- and mid-level system components via machine learning
solution architecture31
Solution Architecture

AnnotationArchitecture

annotateddocuments

LearningArchitecture

automaticannotators

ApplicationArchitecture

solution architecture32
Solution Architecture

Annotation Architecture

AnnotationTool

Learning Architecture

LearningAlgorithms

Trained Taggers

Application Architecture

DocumentRetrieval

PerspectiveTagging

DocumentClustering

Question

Other Taggers

application architecture
Application Architecture

Multi-perspective Classifiers

Document

Clustering

Documents

Annotation Database

Gate NE

CASS

Feature Generators

annotation components
Annotation Components
  • GATE’s ANNIE or MITRE Alembic
    • Tokenization, sentence-finding
    • Part-of-speech tagging
    • Name finding
    • Coreference resolution
  • CASS partial parser
  • SMART IR engine
  • Feature Generators
learning architecture
Learning Architecture

Evaluation

Training

Data

Weka Learner

Weka Learner

Annotation Database

Gate NE

CASS

Feature Generators

learning tasks
Learning: Tasks
  • Identify subjective phrases
  • Identify nested sources
  • Discriminate Facts and Views
  • Classify Opinion Strength
learning features
Learning: Features
  • Name recognition
  • Syntactic features
  • Lists of words
  • Contextual features
  • Density
annotation architecture
Annotation Architecture

TopicDocuments

GateAnnotationTool

HumanAnnotators

Gate XML

MPQADatabase

annotation tool gate
Annotation Tool (GATE)
  • Move headers and original markup to standoff annotation database
  • Initialize document annotations
    • Initial sources and speech events
  • Verify human annotations
    • Check id existence
    • Check attribute consistency
data formats
Data Formats
  • Gate XML Format
    • standoff
    • structured
  • MPQA Annotation Format
    • standoff
    • flat
  • Machine Learning Formats (e.g., ARFF)
gate xml format
Gate XML Format

StartNode=“215” EndNode=“228”>

strength

low

source

w,foreign-ministry

mpqa annotation format
MPQA Annotation Format

id span type name content

42 215,228 string MPQA-agent id=“foreign-ministry”

end user evaluation goal
End-User Evaluation: Goal
  • Establish framework for evaluating tasks that would be of direct interest to analyst users
  • Do an example evaluation
manual clustering
Manual Clustering
  • Human exploratory effort
  • MPQA participants manually cluster documents from 1-2 topics
  • Analyze basis for cluster
user task topic
User Task: Topic
  • U1: User states topic of interest and interacts with IR system
  • S1: System retrieves set of “relevant” documents along with their perspective annotations
example topic
Example: Topic
  • U1: 2002 election in Zimbabwe
  • S1: System returns
    • 03.47.06-11142 Mugabe confident of victory in
    • 04.33.07-17094 Mugabe victory leaves West in
    • 05.22.13-11526 Mugabe says he is wide awake
    • 06.21.57-1967 Mugabe predicts victory
    • 06.37.20-8125 Major deployment of troops
    • 06.47.23-22498 Zambia hails results
user task question
User Task: Question
  • U2: User states particular perspective question on topic.
  • Question should
    • identify source type (eg, governments, individuals, writers) of interest.
    • Be a yes/no (or pro/con) question for now
example question
Example: Question
  • Give Range of perspective: national government,groups of governments:
  • Was the election process fair, valid, and free of voter intimidation?
user task question response
User Task: Question Response
  • S2:System clusters documents
    • based on question,text,annotations
    • goal:group together documents with same answer and perspective (including expressive content).
    • System,for now, does not attempt to label each group with specific answers.
  • Target a small number of clusters (4?)
example question response
Example:Question Response
  • Cluster 1
    • 07.20.20-11694
    • 08.12.40-1611
    • 08.15.19-23507
    • 09.35.06-27851
    • 13.10.41-18948
  • Cluster 2
    • 12.08.27-27397
    • 13.44.36-19236
    • 04.33.07-17094
    • 05.22.13-11526
  • Cluster 3
    • 06.47.23-22498
    • 06.51.18-1222
    • 06.56.31-3120
    • 07.16.31-13271
user task cluster feature
User Task: Cluster Feature
  • U3: User states constraints on clustered documents or segments.
    • These might be geographic, date, ideological, political, religous
  • S3: System shows subclusters or highlighted documents
example cluster by features
Example: Cluster by Features
  • U3: Highlight governments by regions
  • S3: System shows docs with African governments opinions in red, North American in blue, European in green, Asian in purple. Multicolored if docs have more than one source
user task results
User Task: Results
  • U4: User gets impression (visual or statistical) whether constraints match clusters.
  • Easy visualization of exceptions
example results
Example: Results
  • User sees that the
    • Red docs (African) are mostly in one cluster,
    • Blue and green (NA and EU) in another
    • Purple docs are scattered in both clusters.
document collection
Document Collection
  • Large collection of 270,000 foreign news documents from June, 2001 to May, 2002
  • Almost all FBIS documents with a small number of other relevant docs.
  • From MITRE MiTAP system
document collection features
Document Collection Features
  • English Language
    • 60% FBIS translated
    • 40% source English
  • 20% TV/Radio
  • 5% Identified as editorials
slide57
From day Tue Jan 22 20:13:06 2002
  • Received: from smtpsrv1.mitre.org …
  • From: [email protected]
  • Date: 21 Jan 2002 00:00:00 (EST)
  • Subject: Vietnam Calls for Broader …
  • East Asia
  • China
  • Southeast Asia
  • China
  • Vietnam
  • China
  • ENVIRONMENTHEALTH
  • Vietnamese Minister of Science, Technology and Environment Chu Tuan Nha told the first Asia-Europe Meeting [ASEM] Environment Ministers' Meeting [ASEM EnMM] in Beijing recently that Vietnam always values environmental protection, including prevention of pollution or degradation, bio-diversity protection, and improvement of the environment in industrial zones and in both urban and rural areas.
sample pure text
Sample Pure Text
  • Vietnamese Minister of Science, Technology and Environment Chu Tuan Nha told the first Asia-Europe Meeting [ASEM] Environment Ministers' Meeting [ASEM EnMM] in Beijing recently that Vietnam always values environmental protection, including prevention of pollution or degradation, bio-diversity protection, and improvement of the environment in industrial zones and in both urban and rural areas.
sample meta annotation
Sample Meta-annotation
  • 1 0,0 string meta_media_file sep20020122000034n.html
  • 2 0,0 string meta_media_type text
  • 3 0,0 string meta_scribe Rough'n'Ready v1.1
  • 4 0,0 string meta_title Vietnam Calls for Broader Environmental Protection at ASEM Conference in Beijing
  • 5 0,0 string meta_document_time 2002-01-21
  • 6 0,0 string meta_create_time 2002-01-22T20:13:05
  • 7 0,0 string meta_source Worldwide Open-source News
  • 8 0,0 string meta_description Hanoi Voice of Vietnam News
  • 9 0,0 string meta_reference SEP20020122000034 Hanoi Voice of Vietnam News WWW-Text in Vietnamese 21 Jan 02
  • 10 0,0 string meta_region East Asia
  • 11 0,0 string meta_region China
  • 12 0,0 string meta_subregion Southeast Asia
  • 13 0,0 string meta_subregion China
  • 14 0,0 string meta_country Vietnam
  • 15 0,0 string meta_country China
  • 16 0,0 string meta_topic ENVIRONMENT
  • 17 0,0 string meta_topic HEALTH
topics
Topics
  • About 12 Topic statements.
    • Clause or Sentence
  • 25-50 known relevant docs per topic, with manual perspective annotations.
  • 1-5 Questions per topic
questions
Questions
  • Type of Perspective
    • range of perspective,
    • strongly felt perspective,
    • identify all perspective
  • Issue
    • Direct information
    • Opinion evidence
  • Constraints (pinpoint discrepencies)
evaluation on topic question
Evaluation on Topic/Question
  • Artificially construct 75 doc retrieved set
    • Include the known (25-50) rel docs
    • Add top retrieved docs from SMART
  • System automatically annotates set
  • System clusters based on annotation.
evaluation cont
Evaluation (cont)
  • Evaluate homogeneity of clusters. Compare with
    • Base Case 1: Cluster docs into same number of clusters without any annotations
    • Base Case 2: Cluster docs into same number of clusters based on manual annotations.
evaluation within workshop
Evaluation Within Workshop
  • Evaluation through S2 only
    • No constraints, subclusters
  • Yes/No (Pro/Con) questions only
current status
Current Status
  • Document collection prepared, indexed
  • 8 topics (more coming)
    • 16 questions total
    • 10-40 rel docs per topic (more coming)
ad