annotating topics of opinions l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Annotating Topics of Opinions PowerPoint Presentation
Download Presentation
Annotating Topics of Opinions

Loading in 2 Seconds...

play fullscreen
1 / 42

Annotating Topics of Opinions - PowerPoint PPT Presentation


  • 233 Views
  • Uploaded on

Annotating Topics of Opinions. Veselin Stoyanov Claire Cardie. Talk Overview. Fine-grained sentiment analysis Definitions Examples Opinion topic annotation Definitions Issues Approach and Corpus IA agreement. Background. Sentiment Analysis:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Annotating Topics of Opinions' - Thomas


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
annotating topics of opinions

Annotating Topics of Opinions

Veselin Stoyanov

Claire Cardie

talk overview
Talk Overview
  • Fine-grained sentiment analysis
    • Definitions
    • Examples
  • Opinion topic annotation
    • Definitions
    • Issues
    • Approach and Corpus
    • IA agreement

LREC 2008, Marakech, Morocco

background
Background
  • Sentiment Analysis:

Extraction and representation of attitudes, evaluations, opinions, and sentiment in text.

  • Fine-grained Sentiment Analysis:

At the level of individual expressions of opinions.

LREC 2008, Marakech, Morocco

fine grained vs coarse grained sentiment analysis
Coarse-grained

Sentiment classification

Useful in the product review domain

Fine-grained

Individual expressions of opinions

Multiple opinions per document (even sentence)

The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty.

Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given.

Review 1

Positive

[SThe Australian press] has launched a bitter attack on [TItaly] after seeing their beloved[TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty].

Lippi is preparing his side for the upcoming clash with Ukraine. [SHe] hailed 10-man [TItaly]'s determination to beat Australia and reiterated that the [Tpenalty] was rightly given.

Review 2

Negative

Fine-grained vs. Coarse-grained Sentiment Analysis

LREC 2008, Marakech, Morocco

fine grained opinions example
Fine-grained opinions: Example

The Australian press has launched a bitter attack

on Italy.

LREC 2008, Marakech, Morocco

fine grained opinions example6
Fine-grained opinions: Example

The Australian press has launched a bitter attack

on Italy.

  • Opinion trigger (opinion words)
  • Source (opinion holder)
  • Polarity – positive/negative
  • Strength
  • Topic (target)
  • Definitions differ, but five main components:

LREC 2008, Marakech, Morocco

fine grained opinions example7
Fine-grained opinions: Example

The Australian press has launched a bitter attack

on Italy.

  • Opinion trigger (opinion words)
  • Source (opinion holder)
  • Polarity – positive/negative
  • Strength
  • Topic (target)
  • Definitions differ, but five main components:

launched a bitter attack

LREC 2008, Marakech, Morocco

fine grained opinions example8
Fine-grained opinions: Example

[SThe Australian press] has launched a bitter attack

on Italy.

  • Opinion trigger (opinion words)
  • Source (opinion holder)
  • Polarity – positive/negative
  • Strength
  • Topic (target)
  • Definitions differ, but five main components:

launched a bitter attack

The Australian press

LREC 2008, Marakech, Morocco

fine grained opinions example9
Fine-grained opinions: Example

[SThe Australian press] has launched a bitter attack

on Italy.

  • Opinion trigger (opinion words)
  • Source (opinion holder)
  • Polarity – positive/negative
  • Strength
  • Topic (target)
  • Definitions differ, but five main components:

launched a bitter attack

The Australian press

negative

LREC 2008, Marakech, Morocco

fine grained opinions example10
Fine-grained opinions: Example

[SThe Australian press] has launched a bitter attack

on Italy.

  • Opinion trigger (opinion words)
  • Source (opinion holder)
  • Polarity – positive/negative
  • Strength
  • Topic (target)
  • Definitions differ, but five main components:

launched a bitter attack

The Australian press

negative

high

LREC 2008, Marakech, Morocco

fine grained opinions example11
Fine-grained opinions: Example

[SThe Australian press] has launched a bitter attack

on [TItaly]

  • Opinion trigger (opinion words)
  • Source (opinion holder)
  • Polarity – positive/negative
  • Strength
  • Topic (target)
  • Definitions differ, but five main components:

launched a bitter attack

The Australian press

negative

high

Italy

LREC 2008, Marakech, Morocco

fine grained opinions
Fine-grained opinions
  • Five components
    • Source (opinion holder)
      • e.g. [Bethard et al., 2004] [Choi et al., 2005] [Kim and Hovy, 2006]
    • Opinion trigger (opinion words)
      • e.g. [Yu and Hatzivassiloglou, 2003] [Riloff and Wiebe, 2003]
    • Polarity – positive/negative
      • As above
    • Strength
      • e.g. [Wilson et al. 2004]
    • Topic (target)
      • ????

LREC 2008, Marakech, Morocco

annotating topics of fine grained opinions
Annotating Topics of Fine-grained Opinions
  • Definitions
  • Issues
  • Approach and Corpus
  • IA agreement

LREC 2008, Marakech, Morocco

examples
Examples

(1)[OH John] likes Marseille for its weather and cultural diversity.

(2)[OH Al] thinks that the government should tax gas more in order to curb CO2 emissions.

LREC 2008, Marakech, Morocco

definitions
Definitions

(1)[OH John] likes Marseille for its weather and cultural diversity.

LREC 2008, Marakech, Morocco

definitions16
Definitions

(1)[OH John] likes Marseille for its weather and cultural diversity.

Topic: city of Marseille

  • Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder

LREC 2008, Marakech, Morocco

definitions17
Definitions

(1)[OH John] likes [TOPIC SPAN Marseille] for its weather and cultural diversity.

Topic: city of Marseille

  • Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder
  • Topic span - the closest, minimal span of text that mentions the topic

LREC 2008, Marakech, Morocco

definitions18
Definitions

(1)[OH John] likes [TARGET+TOPIC SPAN Marseille] for its weather and cultural diversity.

Topic: city of Marseille

  • Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder
  • Topic span - the closest, minimal span of text that mentions the topic
  • Target span - the span of text that covers the syntactic surface form comprising the contents of the opinion

LREC 2008, Marakech, Morocco

definitions19
Definitions

(2)[OH Al] thinks that the government should tax gas more in order to curb CO2 emissions.

LREC 2008, Marakech, Morocco

definitions20
Definitions

(2)[OH Al] thinks that [TARGET SPAN the government should tax gas more in order to curb CO2 emissions].

LREC 2008, Marakech, Morocco

definitions21
Definitions

(2)[OH Al] thinks that [TARGET SPAN[TOPIC SPAN? the government] should [TOPIC SPAN? tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]].

LREC 2008, Marakech, Morocco

definitions22
Definitions

(2)[OH Al] thinks that [TARGET SPAN the government should tax gas more in order to curb CO2 emissions].

Context:

(3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.

LREC 2008, Marakech, Morocco

definitions23
Definitions

(2)[OH Al] thinks that [TARGET SPAN the government should [TOPIC SPANtax gas] more in order to curb CO2 emissions].

Context:

(3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.

LREC 2008, Marakech, Morocco

related work
Related Work
  • Product reviews
    • E.g. Kobayashi et al. (2004), Yi et al. (2003), Popescu and Etzioni (2005), Hu and Liu (2004
    • Limit “topics” to mentions of product names, components, and their attributes
    • Lexicon look-up
    • Focused on methods for lexicon acquisition
  • MPQA corpus (Wiebe, Wilson, Cardie, 2004)
    • Fine-grained opinions
    • Topic annotation deemed too difficult
    • Target span annotation is underway
  • Kim & Hovy (2006)
    • Target span extraction using semantic frames
    • Limited evaluation

LREC 2008, Marakech, Morocco

issues in opinion topic identification
Issues in Opinion Topic Identification
  • Multiple potential topics mentioned within a single target span

(2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the government] should [TOPIC SPAN? tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]].

  • Requires context

Topic of an opinion is the entity that comprises the main information goal of the opinion based on the discourse context.

LREC 2008, Marakech, Morocco

issues in opinion topic identification26
Issues in Opinion Topic Identification
  • Opinion topics are not always explicitly mentioned

(4) [OH John] believes the violation of Palestinian human rights is one of the main factors.

Topic: ISRAELI-PALESTINIAN CONFLICT

(5) [OH I] disagree entirely!

LREC 2008, Marakech, Morocco

a coreference approach
A Coreference Approach
  • Hypothesize that the notion of topic coreference will facilitate identification of opinion topics
  • Easier than specifying the topic of each opinion in isolation

Two opinions are topic-coreferent if they share the same opinion topic.

LREC 2008, Marakech, Morocco

opinion topic corpus
Opinion Topic Corpus

(www.cs.pitt.edu/mpqa)

Build on the MPQA corpus:

  • 535 Documents manually annotated for fine-grained opinions
  • No opinion topic annotation
  • Our goal: Add the opinion topic information on top of the existing opinion annotations
  • Created and used a GUI

LREC 2008, Marakech, Morocco

annotation process

List of opinions

to be processed

Set of current

clusters

Document text

Annotation Process

LREC 2008, Marakech, Morocco

annotation process30
Annotation Process

LREC 2008, Marakech, Morocco

annotation process31

fuel tax

Annotation Process

LREC 2008, Marakech, Morocco

interannotator agreement
Interannotator Agreement
  • Annotator 1
    • 150 of the 535 MPQA documents
  • Annotator 2
    • 20 of these 150
  • IAG measures from noun phrase coreference resolution

LREC 2008, Marakech, Morocco

interannotator agreement33
Interannotator Agreement
  • Annotator 1
    • 150 of the 535 MPQA documents
  • Annotator 2
    • 20 of these 150
  • IAG measures from noun phrase coreference resolution

LREC 2008, Marakech, Morocco

baselines
Baselines
  • all-in-one
    • assigns all opinions to the same cluster
  • 1 opinion per cluster
    • assigns each opinion to its own cluster
  • same paragraph
    • opinions in the same paragraph are assigned to the same cluster

LREC 2008, Marakech, Morocco

results
Results
  • Baselines
  • vs. Interannotator agreement

LREC 2008, Marakech, Morocco

questions
Questions?Thank you

Annotation instructions + more information available at:

www.cs.cornell.edu/~ves

LREC 2008, Marakech, Morocco

example
Example

The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty.

Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given.

LREC 2008, Marakech, Morocco

example fine grained opinions
Example – fine-grained opinions

[SThe Australian press] has launched a bitter attack on [TItaly] after seeing [Stheir]beloved[TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty].

Lippi is preparing his side for the upcoming clash with Ukraine. [SHe]hailed 10-man [TItaly]'s determination to beat Australia and reiterated that [Tthe penalty] was rightly given.

LREC 2008, Marakech, Morocco

motivation
Motivation
  • Sentiment analysis: Useful as stand-alone application
    • Product reviews
    • Tracking opinions in the press
    • Flame detection, etc.
  • Opinion information can benefit many NLP applications
    • Multi-Perspective Question Answering

[Stoyanov, Cardie, Litman and Wiebe. AAAI WS 2004] and

[Stoyanov, Cardie and Wiebe. HLT-EMNLP 2005]

    • Opinion-Oriented Information Retrieval
    • Clustering, etc.

LREC 2008, Marakech, Morocco

annotation process40
Annotation Process

LREC 2008, Marakech, Morocco

annotation process41
Annotation Process

LREC 2008, Marakech, Morocco

slide42
`

LREC 2008, Marakech, Morocco