error analysis for learning based coreference resolution
Download
Skip this Video
Download Presentation
Error Analysis for Learning-based Coreference Resolution

Loading in 2 Seconds...

play fullscreen
1 / 45

error analysis for learning-based coreference resolution - PowerPoint PPT Presentation


  • 220 Views
  • Uploaded on

Error Analysis for Learning-based Coreference Resolution. Olga Uryupina 27.05.08. Outline. CR: state-of-the-art and our system Distribution of errors Discussion: possible remedies. Coreference Resolution.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'error analysis for learning-based coreference resolution' - libitha


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
outline
Outline
  • CR: state-of-the-art and our system
  • Distribution of errors
  • Discussion: possible remedies
coreference resolution
Coreference Resolution

„This deal means that Bernard Schwartz can focus most of his time on Globalstar and that is a key plus for Globalstar because Bernard Schwartz is brilliant,“ said Robert Kaimovitz, a satellite communications analyst at Unterberg Harris in New York.

..

Globalstar still needs to raise $ 600 million, and Schwartz said that the company would try..

coreference resolution4
Coreference Resolution

„This deal means that Bernard Schwartz can focus most of his time on Globalstar and that is a key plus for Globalstar because Bernard Schwartz is brilliant,“ said Robert Kaimovitz, a satellite communications analyst at Unterberg Harris in New York.

..

Globalstar still needs to raise $ 600 million, and Schwartz said that the company would try..

coreference resolution5
Coreference Resolution

„This deal means that Bernard Schwartz can focus most of his time on Globalstar and that is a key plus for Globalstar because Bernard Schwartz is brilliant,“ said Robert Kaimovitz, a satellite communications analyst at Unterberg Harris in New York.

..

Globalstar still needs to raise $ 600 million, and Schwartz said that the company would try..

machine learning approaches
Machine Learning Approaches
  • Soon et al (2000)
  • Cardie & Wagstaff (1999)
  • Strube et al. (2002)
  • Ng & Cardie (2001-2004)
  • ACE competition
features soon et al 2000
Features: Soon et al. (2000)
  • Anaphor is a pronoun
  • Anaphor is a definite NP
  • Anaphor is an NP with a demonstrative pronoun („this“,..)
  • Antecedent is a pronoun
  • Both markables are proper names
  • Number agreement
  • Gender agreement
  • Alias
  • Appositive
  • Same surface form
  • Semantic class agreement
  • Distance in sentences
features other approaches
Features: other approaches

Cardie & Wagstaff: 11 Features

Strube et al.: 17 Features (the same standard features + approximate matching (MED))

Ng & Cardie: 53 Features (no improvement on the extended feature set, better results (F=63.4) with manual feature selection)

performance soon et al
Performance: Soon et al.

Soon et al‘s system:

Our reimlementation:

performance soon et al10
Performance: Soon et al.

Learning Curve for C5.0

tricky and easy anaphors
Tricky and easy anaphors

Cristea et al. (2002): state-of-the-art coreference resolution systems have essentially the same performance level

Pronominal anaphora – 80%

Full-scale coreference – 60%

Hypothesis: tricky vs. easy anaphors

our system
Our system

Goal:

Bridge the gap between the theory and the practice:

sophisticated linguistic knowledge + data-driven coreference resolution algorithm

new features
New Features

Different aspects of CR:

  • Surface similarity (122 features)
  • Syntax (64)
  • Semantic Compatibility (29)
  • Salience (136)
  • (Anaphoricity)

More or less sophisticated linguistic theories exist for all these phenomena

evaluation
Evaluation

Methodology

  • Standart dataset (MUC-7)
  • Standard learning set-up
  • Compare to Soon et al. (2001)
performance
Performance

Learning Curve, SVM

error analysis
Error analysis

Different approaches – same performance:

  • Same errors?
  • „Tricky anaphors“? (Cristea et al., 2002)

Extensive error analysis needed!

outline18
Outline
  • CR: state-of-the-art and our system
  • Distribution of errors
  • Discussion: possible remedies
recall errors markables
Recall errors - markables
  • Auxilliary doc parts
  • Tokenization
  • Modifiers
  • Bracketing/labeling
recall errors markables21
Recall errors - markables

.. there was no requirement for tether to be manufactured in a contaminant-free enviroment.

A mesmerizing set.

recall errors pronouns
Recall errors - pronouns

1st pl – reconstructing the group:

The retiring Republican chairman of the House Committee on Science want U.S. Businesses to <..> „We need to make it easier for the private sector..“ Walker said

3rd sg, 3rd pl – (non-)salience:

[The explanation] for the History Channel‘s success begin with its association with another channel owned by the same parent consortium.

recall errors nominal
Recall errors - nominal

Mostly common noun phrases with different heads, WordNet does not help much

.. a report on the satellites‘ findings <..> the abilities of U.S. Reconnaissance technology <..> the use of advanced intelligence-gathering tools <..> Remote-sensing instruments..

precision errors pronouns
Precision errors- pronouns
  • incorrect Parsing/Tagging

Two key vice presidents, [Wei Yen] and Eric Carlson, are leaving to start their own Silicon Valley companies.

  • (non-)salience
  • matching (propagated R)
precision errors nominal
Precision errors - nominal

Mostly same-head descriptions. Possible solutions:

  • modifiers?
  • anaphoricicty detectors?
p errors nominal modifiers
P errors – nominal - modifiers

Idea: „red car“ cannot corefer with „blue car“

Problem: list of mutually incompatible properties?

MUC7 test data:

incompatible modifiers 30

„new“ mod for anaphora 15

compatible modifiers 58

no modifiers 62

p errors nominal dnew
P errors – nominal - dnew

Idea: identify and discard unlikely anaphors

Problem: even a very good detector does not help

outline29
Outline
  • CR: state-of-the-art and our system
  • Distribution of errors
  • Discussion: Possible remedies
discussion errors
Discussion – Errors

Problematic areas:

  • Data
  • Preprocessing modules
  • Features
  • Resolution strategy
discussion data
Discussion - Data
  • bigger corpus
  • more uniform doc selection, text only
  • better definition of COREF
  • better scoring
discussion preprocessing
Discussion - Preprocessing
  • local improvements (e.g. appositions)
  • probabilistic architecture to neutralize errors
discussion features
Discussion - Features
  • feature selection
  • ensemble learning
  • more targeted learning for under-represented phenomena (abbreviations)
discussion resolution
Discussion - Resolution
  • less local: move to the chains level
  • less uniform: specific treatment for different types of anaphors
discussion conclusion
Discussion – Conclusion
  • ML approaches to the Coreference Resolution yield similar performance values
  • Some anaphors are indeed tricky (esp. crucial for precision errors)
  • But some errors can be eliminated within a ML framework
    • improving the training material
    • elaborated integration of preprocessing modules
    • more global resolution strategies
recall errors muc
Recall errors - MUC

Mainly incorrect bracketing

..said <COREF .. MIN=„vice president“>Jim Johannesen, <COREF .. MIN=„vice president“>vice president of site development for McDonald‘s</COREF></COREF>..

Only clear typos etc considered MUC-errors

recall errors propagated p
Recall errors – propagated P

The company also said the Marine Corps has begun testing two of [its radars] as part of a short-range ballistic missile defense program. That testing could lead to an order for the radars.

Crucial for pronouns and indicators for intrasentential coreference

recall errors matching
Recall errors - matching

Mostly ORGANIZATIONs.

Problems:

  • Abbreviations

Federal Communication Commission

FCC

  • Hyphenated names

Ziff-Davis Publishing

Ziff

  • Foreign names

Taiwan President Lee Teng-hui

President Lee

recall errors syntax
Recall errors - syntax

Apposition, copula

Problems:

  • Parsing mistakes
  • Missing constructions

..the venture will become synonymous with JSkyB

  • P/R trade-off

..Kevlar, a synthetic fiber, and Nomex..

Quantitative constructions

.. More than quadruple the three-month daily average of 88,700 shares

precision errors matching
Precision errors - matching

Finer NE analysis could help, but mostly too difficult even for humans:

Loral

Loral Space and Communications Corp

Loral Space

Space Systems Loral

anaphoricity
Anaphoricity

Some markables are not anaphors. We can tell that by looking at them, without any sophisticated coreference resolution.

Poesio & Vieira, Ng & Cardie – try to identify Discourse New entities automatically

Not used for this talk

anaphoricity45
Anaphoricity

Some markables are not anaphors. We can tell that by looking at them, without any sophisticated coreference resolution.

Poesio & Vieira, Ng & Cardie – try to identify Discourse New entities automatically

Not used for this talk

ad