word sense and subjectivity l.
Download
Skip this Video
Download Presentation
Word Sense and Subjectivity

Loading in 2 Seconds...

play fullscreen
1 / 71

Word Sense and Subjectivity - PowerPoint PPT Presentation


  • 142 Views
  • Uploaded on

Word Sense and Subjectivity. Jan Wiebe Rada Mihalcea University of Pittsburgh University of North Texas. Introduction. Growing interest in the automatic extraction of opinions, emotions , and sentiments in text (subjectivity).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Word Sense and Subjectivity' - daphne


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
word sense and subjectivity

Word Sense and Subjectivity

Jan Wiebe Rada Mihalcea

University of Pittsburgh University of North Texas

introduction
Introduction
  • Growing interest in the automatic extraction of opinions,emotions, and sentiments in text (subjectivity)
subjectivity analysis applications
Subjectivity Analysis: Applications
  • Opinion-oriented question answering:How do the Chinese regard the human rights record of the United States?
  • Product review mining:What features of the ThinkPad T43 do customers like and which do they dislike?
  • Review classification:Is a review positive or negative toward the movie?
  • Tracking emotions toward topics over time:Is anger ratcheting up or cooling down toward an issue or event?
  • Etc.
introduction4
Introduction
  • Continuing interest in word sense
    • Sense annotated resources being developed for many languages
      • www.globalwordnet.org
    • Active participation in evaluations such as SENSEVAL
word sense and subjectivity5
Word Sense and Subjectivity
  • Though both are concerned with text meaning, they have mainly been investigated independently
subjectivity labels on senses

S

O

Subjectivity Labels on Senses

Alarm, dismay, consternation – (fear resulting from the awareness of danger)

Alarm, warning device, alarm system – (a device that signals the occurrence of some undesirable event)

subjectivity labels on senses7

S

O

Subjectivity Labels on Senses

Interest, involvement -- (a sense of concern with and curiosity about someone or something; "an interest in music")

Interest -- (a fixed charge for borrowing money; usually a percentage of the amount borrowed; "how much interest do you pay on your mortgage?")

wsd using subjectivity tagging

He spins a riveting plot which

grabs and holds the reader’s interest.

Sense 4

Sense 1?

Sense 4 “a sense of concern

with and curiosity about

someone or something” S

Sense 1“a fixed charge

for borrowing money” O

WSD

System

Sense 1

Sense 4?

The notes do not pay interest.

WSD using Subjectivity Tagging
wsd using subjectivity tagging9
WSD using Subjectivity Tagging

He spins a riveting plot which

grabs and holds the reader’s interest.

S

Sense 4

Sense 1?

Sense 4 “a sense of concern

with and curiosity about

someone or something” S

Sense 1“a fixed charge

for borrowing money” O

Subjectivity

Classifier

WSD

System

Sense 1

Sense 4?

O

The notes do not pay interest.

wsd using subjectivity tagging10
WSD using Subjectivity Tagging

He spins a riveting plot which

grabs and holds the reader’s interest.

S

Sense 4

Sense 1?

Sense 4 “a sense of concern

with and curiosity about

someone or something” S

Sense 1“a fixed charge

for borrowing money” O

Subjectivity

Classifier

WSD

System

Sense 1

Sense 4?

O

The notes do not pay interest.

subjectivity tagging using wsd
Subjectivity Tagging using WSD

Subjectivity

Classifier

He spins a riveting plot which

grabs and holds the reader’s interest.

S O?

O S?

The notes do not pay interest.

subjectivity tagging using wsd12

S Sense 4 “a sense of

concern with and curiosity about someone or something”

OSense 1“a fixed charge

for borrowing money”

Subjectivity Tagging using WSD

Subjectivity

Classifier

He spins a riveting plot which

grabs and holds the reader’s interest.

S O?

Sense 4

WSD

System

O S?

Sense 1

The notes do not pay interest.

subjectivity tagging using wsd13

S Sense 4 “a sense of

concern with and curiosity about someone or something”

OSense 1“a fixed charge

for borrowing money”

Subjectivity Tagging using WSD

Subjectivity

Classifier

He spins a riveting plot which

grabs and holds the reader’s interest.

S O?

Sense 4

WSD

System

O S?

Sense 1

The notes do not pay interest

goals
Goals
  • Explore interactions between word sense and subjectivity
    • Can subjectivity labels be assigned to word senses?
      • Manually
      • Automatically
    • Can subjectivity analysis improve word sense disambiguation?
    • Can word sense disambiguation improve subjectivity analysis? Future work
outline
Outline
  • Motivation and Goals
  • Assigning Subjectivity Labels to Word Senses
    • Manually
    • Automatically
  • Word Sense Disambiguation using Automatic Subjectivity Analysis
  • Conclusions
prior work on subjectivity tagging
Prior Work on Subjectivity Tagging
  • Identifying words and phrases associated with subjectivity
    • Think ~ private state;Beautiful ~ positive sentiment
      • Hatzivassiloglou & McKeown 1997; Wiebe 2000; Kamps & Marx 2002; Turney 2002; Esuli & Sabastiani 2005; Etc
  • Subjectivity classification of sentences, clauses, phrases, or word instances in context
    • subjective/objective; positive/negative/neutral
      • Riloff & Wiebe 2003; Yu & Hatzivassiloglou 2003; Dave et al 2003; Hu & Liu 2004; Kim & Hovy 2004; Etc.
  • Here:subjectivity labels are applied toword senses
outline17
Outline
  • Motivation and Goals
  • Assigning Subjectivity Labels to Word Senses
    • Manually
    • Automatically
  • Word Sense Disambiguation using Automatic Subjectivity Analysis
  • Conclusions
annotation scheme
Annotation Scheme
  • Assigning subjectivity labels toWordNet senses
    • S:subjective
    • O:objective
    • B:both
annotators are given the synset and its hypernym

S

Annotators are given the synset and its hypernym

Alarm, dismay, consternation – (fear resulting form the awareness of danger)

  • Fear, fearfulness, fright – (an emotion experiences in anticipation of some specific pain or danger (usually accompanied by a desire to flee or fight))
subjective sense definition
Subjective Sense Definition
  • When the sense is used in a text or conversation, we expect it to express subjectivity, and we expect the phrase/sentence containing it to be subjective.
objective senses observation
Objective Senses: Observation
  • We don’tnecessarily expect phrases/sentences containing objective senses to be objective
    • Would you actually be stupid enough to pay that rate of interest?
    • Will someone shut that darn alarm off?
  • Subjective, but notdue tointerest or alarm
objective sense definition
Objective Sense Definition
  • When the sense is used in a text or conversation, we don’t expect it to express subjectivity and,if the phrase/sentence containing it issubjective, the subjectivity is due tosomething else.
senses that are both
Senses that are Both
  • Covers both subjective and objective usages
  • Example:

absorb, suck, imbibe, soak up, sop up, suck up, draw, take in, take up – (take in, alsometaphorically;“The sponge absorbs water well”;“She drew strength from the Minister’s Words”)

annotated data
Annotated Data
  • 64 words; 354 senses
    • Balanced subset [32 words; 138 senses]; 2 judges
    • The ambiguous nouns of the SENSEVAL-3 English Lexical Task [20 words; 117 senses]; 2 judges
      • [Mihalcea, Chklovski & Kilgarriff, 2004]
    • Others [12 words; 99 senses]; 1 judge
annotated data agreement study
Annotated Data: Agreement Study
  • 64 words; 354 senses
    • Balanced subset [32 words; 138 senses]; 2 judges
      • 16 words have both S and O senses
      • 16 words do not (8 only S and 8 only O)
      • All subsets balanced between nouns and verbs
      • Uncertain tags also permitted
inter annotator agreement results
Inter-Annotator Agreement Results
  • Overall:
    • Kappa=0.74
    • Percent Agreement=85.5%
inter annotator agreement results27
Inter-Annotator Agreement Results
  • Overall:
    • Kappa=0.74
    • Percent Agreement=85.5%
  • Without the 12.3% cases when a judge is U:
    • Kappa=0.90
    • Percent Agreement=95.0%
inter annotator agreement results28
Inter-Annotator Agreement Results
  • Overall:
    • Kappa=0.74
    • Percent Agreement=85.5%
  • 16 words with S and O senses: Kappa=0.75
  • 16 words with only S or O: Kappa=0.73

Comparable difficulty

inter annotator agreement results29
Inter-Annotator Agreement Results
  • 64 words; 354 senses
    • The ambiguous nouns of the SENSEVAL-3 English Lexical Task [20 words; 117 senses] 2 judges
      • U tags not permitted
      • Even so, Kappa=0.71
outline30
Outline
  • Motivation and Goals
  • Assigning Subjectivity Labels to Word Senses
    • Manually
    • Automatically
  • Word Sense Disambiguation using Automatic Subjectivity Analysis
  • Conclusions
related work
Related Work
  • unsupervised word-sense ranking algorithm of [McCarthy et al 2004]
    • That task:approximate corpus frequencies of word senses
    • Our task: predict a word-sense property (subjectivity)
  • method for learning subjective adjectives of[Wiebe 2000]
    • That task:label words
    • Our task:label word senses
overview
Overview
  • Main idea: assess the subjectivity of a word sense based on information about the subjectivity of
    • a set of distributionally similar words
    • in a corpus annotated with subjective expressions
mpqa opinion corpus
MPQA Opinion Corpus
  • 10,000 sentences from the world press annotated for subjective expressions
      • [Wiebe at al., 2005]
      • www.cs.pitt.edu/mpqa
subjective expressions
Subjective Expressions
  • Subjective expressions: opinions, sentiments, speculations, etc. (private states)expressed in language
examples
Examples
  • His alarm grew.
  • The leaders roundly condemned the Iranian President’s verbal assault on Israel.
  • He would be quite a catch.
  • That doctor is a quack.
preliminaries subjectivity of word w

Annotated

Corpus

(MPQA)

Unannotated

Corpus

(BNC)

Lin 1998

#insts(DSW) in SE - #insts(DSW) not in SE

#insts (DSW)

subj(w) =

DSW = {dsw1, …, dswj}

Preliminaries: subjectivity of word w
subjectivity of word w

Annotated

Corpus

(MPQA)

DSW = {dsw1, …, dswj}

Subjectivity of word w

Unannotated

Corpus

(BNC)

#insts(DSW) in SE - #insts(DSW) not in SE

#insts (DSW)

subj(w) =

[-1, 1] [highly objective, highly subjective]

subjectivity of word w38

Annotated

Corpus

(MPQA)

dsw1inst1

dsw1inst2

dsw2inst1

Unannotated

Corpus

(BNC)

+1

-1

+1

+1 -1 +1

subj(w) =

= 1/3

3

DSW = {dsw1,dsw2}

Subjectivity of word w
subjectivity of word sense w i

Annotated

Corpus

(MPQA)

dsw1inst1

dsw1inst2

dsw2inst1

+sim(wi,dsw1) - sim(wi,dsw1) + sim(wi,dsw2)

subj(wi) =

2 * sim(wi,dsw1) + sim(wi,dsw2)

Subjectivity of word sense wi

Rather than 1, add or subtract

sim(wi,dswj)

+sim(wi,dsw1)

[-1, 1]

-sim(wi,dsw1)

+sim(wi,dsw2)

method step 1
Method –Step 1
  • Given word w
  • Find distributionally similar words [Lin 1998]
    • DSW = {dswj | j = 1 .. n}
    • Experiment with top 100 and 160
method step 242
Method – Step 2
  • Find the similarity between each word sense and each distributionally similar word
  • wnss can be any concept-based similarity measure between word senses
    • we use Jiang & Conrath 1997
method step 243
Method – Step 2
  • Find the similarity between each word sense and each distributionally similar word
  • wnss can be any concept-based similarity measure between word senses
    • we use Jiang & Conrath 1997
method step 244
Method – Step 2
  • Find the similarity between each word sense and each distributionally similar word
  • wnss can be any concept-based similarity measure between word senses
    • we use Jiang & Conrath 1997
method step 245
Method – Step 2
  • Find the similarity between each word sense and each distributionally similar word
  • wnss can be any concept-based similarity measure between word senses
    • we use Jiang & Conrath 1997
method step 246
Method – Step 2
  • Find the similarity between each word sense and each distributionally similar word
  • wnss can be any concept-based similarity measure between word senses
    • we use Jiang & Conrath 1997
method step 3
Method –Step 3

Input:word sense wi of word w

DSW = {dswj | j = 1..n}

sim(wi,dswj)

MPQA Opinion Corpus

Output:subjectivity score subj(wi)

method step 348
Method –Step 3

totalsim = #insts(dswj) * sim(wi,dswj)

subj = 0

for each dswj in DSW:

for each instance k in insts(dswj):

if k is in a subjective expression:

subj += sim(wi,dswj)

else:

subj -= sim(wi,dswj)

subj(wi) = subj / totalsim

method optional variation
Method – Optional Variation

if k is in a subjective expression:

subj += sim(wi,dswj)

else:

subj -= sim(wi,dswj)

w1 dsw1 dsw2 dsw3

w2 dsw1 dsw2 dsw3

w3 dsw1 dsw2 dsw3

“Selected”

evaluation
Evaluation
  • Calculate subjscores for all word senses, and sort them
  • While 0 is a natural candidate for division between S and O, we perform the evaluation for different thresholds in [-1,+1]
  • Calculate the precision of the algorithm at different points of recall
evaluation51
Evaluation
  • Automatic assignment of subjectivity for 272 word senses (no DSW instances for 82 senses)
  • Baseline: random selection of S labels
      • Number of assigned S labels matches number of S labels in the gold standard (recall = 1.0)
evaluation precision recall curves
Evaluation: precision/recall curves

Number of distri-butionally similar

words = 160

evaluation53
Evaluation
  • Break-even point
      • Point where precision and recall are equal
outline54
Outline
  • Motivation and Goals
  • Assigning Subjectivity Labels to Word Senses
    • Manually
    • Automatically
  • Word Sense Disambiguation using Automatic Subjectivity Analysis
  • Conclusions
overview55
Overview
  • Augment an existing WSD system with a feature reflecting the subjectivity of the context of the ambiguous word
  • Compare the performance of original and subjectivity-aware WSD systems
  • The ambiguous nouns of the SENSEVAL-3 English Lexical Task
  • SENSEVAL-3 data
original wsd system
Original WSD System
  • Integrates local and topical features:
      • Local: context of three words to the left and right, their part-of-speech
      • Topical: top five words occurring at least three times in the context of a word sense
      • [Ng & Lee, 1996], [Mihalcea, 2002]
  • Naïve Bayes classifier
      • [Lee & Ng, 2003]
automatic subjectivity classifier
Automatic Subjectivity Classifier
  • Rule-based automatic sentence classifier from [Wiebe & Riloff 2005]
  • Included in OpinionFinder;available at:
    • www.cs.pitt.edu/mpqa/
subjectivity tagging for wsd

S

“interest”

Sentencei

“interest”

Sentencej

Subjectivity Tagging for WSD

Used to tag sentences of the SENSEVAL-3 data that contain

target nouns

Subjectivity

Classifier

O

S

“atmosphere”

Sentencek

wsd using subjectivity tagging59

S

“interest”

Sentencei

Sense 4

Sense 1

Subjectivity

Classifier

S, O, or B

Original

WSD System

Subjectivity

Aware WSD

System

Sense 1 “a sense of concern with and curiosity about someone or something”

Sense 4 “a fixed charge for

borrowing money”

WSD using Subjectivity Tagging
words with s and o senses
Words with S and O Senses

<

<

<

<

<

<

<

=

<

=

4.3% error reduction; significant (p < 0.05 paired t-test)

conclusions
Conclusions
  • Can subjectivity labels be assigned to word senses?
    • Manually
      • Good agreement; Kappa=0.74
      • Very good when uncertain cases removed; Kappa=0.90
    • Automatically
      • Method substantially outperforms baseline
      • Showed feasibility of assigning subjectivity labels to the fine-grained level of word senses
conclusions63
Conclusions
  • Can subjectivity analysis improve word sense disambiguation?
    • Improves performance, but mainly for words with both S and O senses (4.3% error reduction; significant (p < 0.05))
    • Performance largely remains the same or degrades for words that don’t
    • Assign subjectivity labels to WordNet; WSD system should consult WordNet tags to decide when to pay attention to the contextual subjectivity feature.
refining wordnet
Refining WordNet
  • Semantic Richness
  • Find inconsistencies and gaps
    • Verbassault – attack, round, assail, last out, snipe, assault (attack in speech or writing) “The editors of the left-leaning paper attacked the new House Speaker”
    • But no sense for the noun as in “His verbal assault was vicious”
observation mpqa corpus
Observation MPQA corpus
  • Corpus somewhat noisy for our task
      • MPQA annotates subjective expressions
      • Objective senses can appear in subjective expressions
  • Hypothesis: subjective senses tend to appear more often in subjective expressions than objective senses do, and so the appearance of words in subjective expressions is evidence of sense subjectivity
wsd using subjectivity tagging67
WSD using Subjectivity Tagging

Hypothesis: instances of subjective senses are more likely to be in subjective sentences, sosentence subjectivity is an informative feature for WSD of words with both subjective and objective senses

subjective sense examples
Subjective Sense Examples
  • He was boilingwith anger

Seethe, boil – (be in an agitated emotional state; “The customer was seething with anger”)

    • Be – (have the quality of being; (copula, used with an adjective or a predicate noun); “John is rich”; “This is not a good answer”)
subjective sense examples69
Subjective Sense Examples
  • What’s thecatch?

Catch – (a hidden drawback; “it sounds good but what’s the catch?”)

Drawback – (the quality of being a hindrance; “he pointed out all the drawbacks to my plan”)

  • That doctor is a quack.

Quack – (an untrained person who pretends to be a physician and who dispenses medical advice)

    • Doctor, doc, physician, MD, Dr., medico
objective sense examples
Objective Sense Examples
  • The alarmwent off

Alarm, warning device, alarm system – (a device that signals the occurrence of some undesirable event)

    • Device – (an instrumentality invented for a particular purpose; “the device is small enough to wear on your wrist”; “a device intended to conserve water”
  • The water boiled

Boil – (come to the boiling point and change from a liquid to vapor; “Water boils at 100 degrees Celsius”)

    • Change state, turn – (undergo a transformation or a change of position or action)
objective sense examples71
Objective Sense Examples
  • He sold his catch at the market

Catch, haul – (the quantity that was caught; “the catch was only 10 fish”)

    • Indefinite quantity – (an estimated quantity)
  • The duck’s quackwas loud and brief

Quack – (the harsh sound of a duck)

    • Sound – (the sudden occurrence of an audible event)