do summaries help a task based evaluation of multi document summarization l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Do Summaries Help? A Task-Based Evaluation of Multi-Document Summarization PowerPoint Presentation
Download Presentation
Do Summaries Help? A Task-Based Evaluation of Multi-Document Summarization

Loading in 2 Seconds...

play fullscreen
1 / 23

Do Summaries Help? A Task-Based Evaluation of Multi-Document Summarization - PowerPoint PPT Presentation


  • 337 Views
  • Uploaded on

Do Summaries Help? A Task-Based Evaluation of Multi-Document Summarization Kathleen McKeown , Rebecca J. Passonneau David K . Elson , Ani Nenkova , Julia Hirschberg Columbia University SIGIR 2005 INTRODUCTION

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Do Summaries Help? A Task-Based Evaluation of Multi-Document Summarization' - emily


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
do summaries help a task based evaluation of multi document summarization

Do Summaries Help?A Task-Based Evaluation of Multi-Document Summarization

Kathleen McKeown , Rebecca J. Passonneau

David K . Elson , Ani Nenkova , Julia Hirschberg

Columbia University

SIGIR 2005

introduction
INTRODUCTION
  • Newsblaster, a system that provides an interface to browse the news, featuring multi-document summaries of clusters of articles on the same event
  • key components of Newsblaster:
    • Article clustering
    • Event cluster summarization
    • User interface
methods
METHODS
  • In the Experiment, subjects are asked to write a report using a news aggregator as a tool.
  • Each subject was asked to perform four 30-minute fact gathering scenarios using a Web interface.
  • Each scenario involved answering three related questions about an issue in the news.
methods5
METHODS

The four tasks were:

  • The Geneva Accord (日內瓦協定) in the Middle East
  • Hurricane Ivan’s (伊凡颶風) effects
  • Conflict in the Iraqi city of Najaf
  • Attacks by Chechen (車臣) separatists in Russia
methods7
METHODS
  • Subjects were given a Web page that we constructed as their sole resource
  • The page contained four document clusters, two of which were centrally related to the topic at hand, and two of which were peripherally related.
  • Each cluster contained, on average, ten articles.
methods8
METHODS

Four summary condition levels:

  • Level 1: No summaries
  • Level 2: One-sentence summary for each article, one-sentence summary for each entire cluster
  • Level 3: Newsblaster multi-document summary for each cluster
  • Level 4: Human multi-document summary for each cluster
experiment
Experiment
  • 45 subjects for three studies
  • Study A: 21 subjects wrote reports for two scenarios each in two summary conditions: Level 3 and Level 4.
  • Study B: 11 subjects wrote reports for all four scenarios, using Summary Level 2.
  • Study C: 13 subjects wrote reports for all four scenarios, using Summary Level 1.
scoring
Scoring
  • Use the Pyramid method for evaluation
  • For example, to score a report from the human summary, we constructed the pyramid using reports created using all other conditions, plus the reports written by otherpeople with human summaries.
  • If there are n reports, then there will be n levels in the pyramid. The top level will contain those facts that appear in all articles
scoring11
Scoring
  • Let Tj refer to the jth tier in a pyramid of facts. If the pyramid has n tiers, then Tn is the top-most tier and T1, the bottom-most. The score for a report with X facts is:
  • where j is equal to the index of the lowest tier an optimally informative report will draw from.
scoring13
Scoring
  • It has been observed that report length has a significant effect on evaluation results
  • We restricted the length of reports to be no longer than one standard deviation above the mean, and we truncated all question answers to a length of eight content units, which was the third quartile of lengths of all answers.
results
RESULTS

We measured in three ways:

  • 1. By scoring the reports and comparing scores across summary conditions
  • 2. By comparing user satisfaction per summary condition
  • 3. By comparing where subjects preferred to draw report content from, measured by counting the citations they inserted following each extracted fact.
content score comparison
Content Score Comparison
  • Differences between the scores

are not significant (p=0.3760 from ANOVA analysis) in Table 1.

  • This may have been due to the fact that the event clusters for Geneva contained more editorials with less “hard” news, while the clusters for Hurricane Ivan contained more breaking news reports.
content score comparison17
Content Score Comparison
  • After removing the Geneva Accord scenario scores, Newsblaster summaries is significantly better than documents only
  • The differences between Newsblaster and minimal or human summaries are not significant
discussion
DISCUSSION
  • When we developed Newsblaster, we speculated that summaries could help users find necessary information in the following ways:
  • 1. they may find the information they need in the summaries themselves, thus requiring them to read less of the full articles
  • 2. the summaries may link them directly to the relevant articles and positions within the articles where the relevant

information occurs

discussion21
DISCUSSION
  • There were two problems
  • First, the interface for Summary Level 2 identified individual articles with a title and a one sentence summary; but Level 3 only had titles for each article.
  • Second, the interface for Summary Level 2 shows the list of individual articles on the same Web page for the cluster; but Level 3 shows the summary and cluster title on the same page and requires the subject to click on cluster title to see the list of individual articles.
discussion22
DISCUSSION
  • Another problem that we noted was that reports written by subjects were of widely varying length.
  • Reports varied from 102 words to 1525 words. We adjusted for this by truncating reports.
  • Lengthy reports tended to have more duplication of facts, which clearly makes for less effective reports.
  • The impact of truncating reports requires follow-up study
conclusions
CONCLUSIONS
  • Our answer to the question, Do Summaries Help?, is clearly yes. Our results show that subjects produce better quality reports using a news interface with Newsblaster summaries than with no summaries.
  • Users are also more satisfied with multi-document summaries than with minimal

one-sentence summaries such as those used by commercial online news systems.

  • Interface design, report length, and scenario design effect on task completion