towards automated related work summarization n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Towards Automated Related Work Summarization PowerPoint Presentation
Download Presentation
Towards Automated Related Work Summarization

Loading in 2 Seconds...

play fullscreen
1 / 70

Towards Automated Related Work Summarization - PowerPoint PPT Presentation


  • 108 Views
  • Uploaded on

Towards Automated Related Work Summarization. Cong Duy Vu Hoang July 2010. Outline. Introduction Previous Studies Data Manual Analysis Proposed System Experiments & Results Future Work Conclusion. Introduction. Scenario :. Prior community knowledge. relate.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Towards Automated Related Work Summarization' - clodia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
outline
Outline
  • Introduction
  • Previous Studies
  • Data
  • Manual Analysis
  • Proposed System
  • Experiments & Results
  • Future Work
  • Conclusion
introduction
Introduction
  • Scenario:

Prior

community

knowledge

relate

a research topic/problem of interest

Scholars

return

A very long list of related works

introduction1
Introduction
  • Example: use topic “multi-document summarization”, search engines may return a long list of hits

Read through all of them

is tedious and time-consuming

introduction2
Introduction
  • Motivation

Prior

community

knowledge

relate

a research topic/problem of interest

Scholars

return

How to re-organize this list as

a compact related work summary?

A very long list of related works

introduction3
Introduction
  • Motivation
    • I envision an NLP application that assists in creating a related work summary.
    • I propose the task of related work summarization
      • topic-biased, multi-document summarization problem
        • Input: a target research problem
        • Output: a related work summary needs to be drafted
    • This work examines the feasibility and possibility of the proposed task.
introduction4
Introduction
  • Related work summarization is significantly different from traditional summarization
    • Limited to the domain of scientific discourse
    • The output summary follows a specific structure of example related work sections
    • Evaluation is non-trivial, requires special evaluation metrics
previous studies
Previous Studies
  • There are no existing studies on this specific task!
  • Single-document scientific article summarization
    • (Luhn,1958; Baxendale,1958; Edmundson,1969)  surface features, extracts of technical documents
    • (Schwartz and Hearst, 2006)  citation texts, key concepts of bioscience texts
    • (Mei and Zhai, 2008; Qazvinian and Radev, 2008)  citation texts, computational linguistics
  • The iOPENER project works towards automated creation of technical surveys given a research topic.
    • (Mohammad et al., 2009)  structure of technical surveys using citation texts, multiple article summarization
    • (Qazvinian and Radev, 2010)  background information for generating technical surveys
previous studies1
Previous Studies
  • Technical book summarization
    • (Mihalcea and Ceylan, 2007)  novel features based on text segmentation for summarization
  • Rhetorical analysis of scientific texts
    • (Teufel, 1999;Teufel and Moens, 2002)  argumentative zoning (AZ), computational linguistics
    • (Teufel et al., 2009)  AZ, chemistry domain
    • (Angrosh et al., 2010)  rhetorical classification scheme for related work section
  • Literature Review Generation
    • (Jaidka et al., 2010)  discourse analysis of literature review, decomposition
slide10
Data
  • Data Construction
    • To create a data set for analysis & evaluation
    • Randomly collected 20 articles in different major conference proceeding in NLP & text processing
      • ACL(6), EMNLP(1), NAACL(5), COLING(4), and SIGIR(4)
    • Extract related work sections and its referenced articles
    • Pre-processing (PDF-to-TXT conversion, sentence boundary) & manual error correction
    • Named RWSData
slide11
Data
  • Data Statistics

No, RW, RA, SbL, and WbL are labeled as (N)umber(o)f, (R)elated (W)orks, (R)eferenced (A)rticles, (S)entence-(b)ased (L)ength of, and (W)ord-(b)ased (L)ength of, respectively.

manual analysis
Manual Analysis
  • Objective:
    • To study characteristics of related work summaries
    • To deconstruct actual related work summaries
      • To gain a deeper insight on how they are structured and authored, from both rhetorical and content levels as well as on the surface, lexical level.
      • Towards efficient strategies for summarization and generation.
manual analysis1
Manual Analysis
  • Formal definition
    • a related work summary (RWS) is a text summary
      • covers stuffs of previous works that are relevant to current work
        • particularly indicating particular aspects of interest (e.g. evaluation, results, experiments, …)
      • mentions the similarities and dissimilarities about particular aspects relevant a topic among previous works
manual analysis2
Manual Analysis
  • Position: two possible positions
    • within the introduction section or the section on its own at the beginning of the article immediately after the Introduction section
      • give a strong overview about previous work
    • before the Conclusion section
      • a relatively short outline of previous studies and adequate comparisons between the technical content of the current study and previous studies
manual analysis3
Manual Analysis

Repeated for other topics

Possible to generate automatically

Extremely hard to generate automatically

A topical structure of a related work summary

slide16

An illustrating example about structure of a related work summary

General topic

claim

Topic 1

Other description & results

Description & Results

Topic 2

Proposed statement

Description & Results

related work summary structure

Paraphrase evaluation

Subjective manual evaluation

Through improving performance of particular tasks (e.g. question answering, machine translation)

Related Work Summary - Structure
  • RWS is topic-biased summary following a topic hierarchy tree

A topic hierarchy tree for previous example

manual analysis4
Manual Analysis
  • Annotation of topical information for RWSData
    • Each related work summary associated with topic hierarchy tree
    • Statistics

TS: tree depth

TD: tree size

manual analysis the decomposition
Manual Analysis - The Decomposition
  • Decomposition is a way to understand the human process in creating a related work summary.
  • Help answer motivated questions:
    • A summary is created by human cut-and-paste operations?
    • Which components in the summary come from the original articles and where in the original document they come from?
      • Levels: words, phrases, clauses, or even sentences
    • How such components are constructed? Revisions?
manual analysis the decomposition1
Manual Analysis - The Decomposition
  • Previous studies proposed automatic decomposition algorithm
    • (Jing et al., 1999) for news articles
    • (Ceylan et al., 2009) for books
  • Becomes non-trivial for multi-document summarization
    • Initially, I approach a manual decomposition for RW summarization
manual analysis the decomposition2
Manual Analysis - The Decomposition
  • The Alignment
  • Human Revisions in Related Work Creation
manual analysis the alignment
Manual Analysis - The Alignment

Example of the alignment process

manual analysis the alignment1
Manual Analysis - The Alignment
  • I observed four categories of RWS sentences:
    • RWS1: (XX, 2000) ... - a summary of an aspect mentioned in referenced article with respect to a specific topic.
      • (Barzilay and McKeown 2001) evaluated their paraphrases by asking judges …
    • RWS2: Topic (XX, 2000) ... - summary of a topic.
      • Supervised approaches such as (Black et al. 1998) have used clustering …
    • RWS3: Fact or Opinion (XX, 2000) ... - evidence-based reference.
      • Co-training (Riloff and Jones, 1999; Collins and Singer, 1999) begins with ...
    • RWST: template-based summary, focus mainly on something about survey paper, dataset, metric, tool, and so on.
      • Sebastiani’s survey paper [23] provides an overview ...
manual analysis the alignment2
Manual Analysis - The Alignment
  • 5 sets chosen for the alignment
  • RWS summary is
    • not necessary to say everything about the referenced articles
    • but just refer to some specific aspects (e.g. of methods, results, evaluation ... )
manual analysis the alignment3
Manual Analysis - The Alignment
  • Relevant information can appear at various positions in original documents
    • Title and Abstract, Introduction, Body (usually Experiments and Results), Conclusion
manual analysis revisions
Manual Analysis - Revisions
  • Sentence Reduction
    • Text fragment 1: ... substituted each set of candidate paraphrasesinto between 2-10 sentenceswhich contained the original phrase.
    • RWSsentence: (Bannard and Callison-Burch 2005) replaced phraseswith paraphrasesin a number of sentences...
manual analysis revisions1
Manual Analysis - Revisions
  • Sentence Combination
    • Text fragment 1: ... substitutedeach set of candidate paraphrasesinto between 2-10 sentenceswhich contained the original phrase.
    • Text fragment 2: ... had two native English speakers produce judgmentsas to whether the new sentences preservedthe meaningof the original phraseand as to whether they remained grammatical.
    • RWSsentence: (Bannard and Callison-Burch 2005) replacedphraseswith paraphrasesin a number of sentencesand asked judgeswhether the substitutions“preserved meaningand remained grammatical”.
manual analysis revisions2
Manual Analysis - Revisions
  • Sentence Combination
    • Text fragment 1: ... to preserveboth meaning and grammaticality.
    • RWSsentence: ... “preservedmeaning and remained grammatical”.
manual analysis revisions3
Manual Analysis - Revisions
  • Lexical Paraphrasing
    • Text fragment 1: ... substitutedeach set of candidate paraphrases into between 2-10 sentences which contained the original phrase.
    • RWSsentence: (Bannard and Callison-Burch 2005) replacedphrases with paraphrases in a number of sentences ...
manual analysis revisions4
Manual Analysis - Revisions
  • Generalization/Specification
    • Text fragment 1: We present an unsupervised learning algorithm that mines large text corporafor patterns that express implicit semantic relations.
    • RWSsentence: (Turney 2006a) presents an unsupervised algorithm for mining the Webfor patterns expressing implicit semantic relations.
manual analysis revisions5
Manual Analysis - Revisions
  • All of the above revisions are generally not used alone but usually combined together to construct sentences in a RWS
  • Dealing with all the above revisions for RWSsummarization is very hard, especially in two revisions:
    • lexical paraphrasing
    • generalization/specification
  • Consider the remaining revisions only!
manual analysis related work representation
Manual Analysis – Related Work Representation
  • To examine how to generate and represent a complete RWS
  • Used another data set (RWSData-Sub) including 30 articles for this analysis.
  • There are two main factors which reflect related work summary representation
    • Topic transition
    • Local coherence
topic transition
Topic transition
  • The observation on chosen data set reveals that there are two types of topic representation for related work summaries
    • Type 1: using transition sentences to connect between topic nodes
      • (23/30-77%)
    • Type 2: representing topic nodes as topic titles
      • (07/30-23%)
  • Type 2 representation sometimes used in the case that there exists a combination of different research problems relevant to a specific research topic. (See some examples)
topic transition1

0

Topic transition

1

2

3

4

0

3

1

4

2

Type 1

topic transition2

0

Topic transition

1

2

3

0

2

1

3

Type 2

topic transition3
Topic Transition
  • For “Type 1” representation:
    • Makes related work natural but:
      • Non-trivial for automatic generation because of lack of topic discourse information, e.g. “contrast”, “elaboration” b/w topic nodes
  • For “Type 2” representation:
    • Not as natural as Type 1 but:
      • Seems to be easy for automatic generation
local coherence
Local Coherence
  • Local coherence
    • The syntactic realization of discourse entities and transitions between focused entities
    • News summaries: entity = mention to people
      • RWS: entity = mention to citations
    • My analysis reveals that there are 14 patterns for mention to citations in RWSes
local coherence1
Local Coherence
  • Statistics 1

Statistics for 14 patterns over the RWSData-Sub data set.

local coherence2
Local Coherence
  • Statistics 2

Statistics for 14 patterns that appear in each type of topic transition representation over the RWSData-Sub dataset

task formulation
Task Formulation

RW: related work

A set of articles

[]

RW Summarizer

User

A desired length

[,]

[]

A RW summary

Topic hierarchy tree

assumption

a motivating example
A Motivating Example

A related work section extracted from “Bilingual Topic Aspect Classification with A few Training Examples” (Wu et al., 2008)

the proposed approach
The Proposed Approach

For leaf nodes

For internal nodes

The ReWoS architecture, Decision edges are labeled as (T)rue, (F)alse or (R)elevant.

the proposed approach1
The Proposed Approach
  • Pre-Processing
    • Based on heuristic rules of sentence length and lexical clues
      • Sentences with token-based length is too short (<7) or too long (>80)
      • Sentences referring to future tenses
      • Sentences containing obviously redundant clues such as: “in the section ...”, “figure XXX shows ...”, “for instance” …
the proposed approach2
The Proposed Approach
  • Agent-based rule
    • Attempts to distinguish whether the sentence describes an author’s own work or not.
    • Based on the presence of tokens that signals work done by the author, such as “we”, “our”, “us”, “this approach”, and “this method” …
    • Says that if a sentence does not satisfy this rule, route for GCSum, otherwise for SCSum
general content summarization gcsum
General Content Summarization (GCSum)
  • The objective of GCSum is to extract sentences containing useful background information on the topics of the internal node in focus.
general content summarization gcsum1
General Content Summarization (GCSum)

General content

informative

indicative

  • Text classification is a task that assigns a certain number of pre-defined labels for a given text.
  • Statistical machine translation (SMT) seeks to develop mathematical models of the translation process whose parameters can be automatically estimated from a parallel corpus.
  • Many previous studies have approached the problem of mono-lingual text classification.
  • This paper refers to the problem of sentiment analysis.
general content summarization gcsum2
General Content Summarization (GCSum)
  • Informative sentences
    • Give detail on a specific aspect of the problem, e.g. definitions, purpose or application of the topic
  • Indicative sentences
    • simpler, inserted to make the topic transition explicit and rhetorically sound
  • Summarization issue
    • Given a topic:
      • For indicative sentences, using pre-defined templates
      • For informative sentences, extract from input articles
general content summarization gcsum3
General Content Summarization (GCSum)

GCSum first checks the subject of each candidate sentence, filtering ones whose subjects do not contain at least one topic keyword. (Subject-based rule)

Or GCSum checks whether stock verb phrases (i.e., “based on”, “make use of” and 23 other patterns) are used as the main verb. (Verb-based rule)

Or GCSum checks for the presence of at least one citation – general sentences may list a set of citations as examples. (Citation-based rule)

Importantly note that if cannot find out any informative sentences from input articles, generate indicative sentences instead!

general content summarization gcsum4
General Content Summarization (GCSum)
  • Topic relevance computation (GCSum)
    • ranks sentences based on keyword content
    • states that the topic of an internal node is affected by its surrounding nodes – ancestor, descendants and others

- scoreS is the final relevance score

- scoreSQA, scoreSQ, and scoreSQR mean the component relevance score of the sentence S with respect to the ancestor, current or other remaining nodes,respectively.

general content summarization gcsum5
General Content Summarization (GCSum)
  • Topic relevance computation (GCSum)

ancestors

1

The maximum number of sentences for each intermediate node is 2-3.

itself

4

5

others

2

3

6

7

The linear combination: S’( ) = S( ) + S( ) – S(5 x )

4

4

1

ancestors

itself

others

general content summarization gcsum6
General Content Summarization (GCSum)
  • To obtain each component relevance score, we employ TF×ISF relevance computation
specific content summarization scsum
Specific Content Summarization (SCSum)
  • Sentences that are marked with author-as-agent are input to the Specific Content Summarization (SCSum) module.
  • SCSum aims to extract sentences that contain detailed information about a specific author’s work that is relevant to the input leaf nodes’ topic.
specific content summarization scsum1
Specific Content Summarization (SCSum)
  • Topic relevance computation (SCSum)

1

ancestors

Initially, the number of sentences for each leaf node is assigned equivalently.

The relevance score is computed using the formula similar to GCSum presented earlier.

4

5

siblings

2

3

6

7

itself

The linear combination: S’( ) = S( + ) + S( ) – S( )

4

2

3

2

1

itself

siblings

ancestors

specific content summarization scsum2
Specific Content Summarization (SCSum)
  • Context modeling
    • Motivation: single sentences occasionally do not contain enough context to clearly express the idea mentioned in original articles
    • Try to use the contexts to increase the confidence of agent-based sentences

topic

score(contexts)

final_score(sentence)

score(sentence)

+

scsum context modeling
SCSum - Context modeling

Example extracted from (Bannard and Callison-Burch 2005)

*** Weevaluated the accuracy of each of the paraphrases that was extracted from the manually aligned data, as well as the top ranked paraphrases from the experimental conditions detailed below in Section 3.3.

*** Because the accuracy of paraphrases can vary depending on context, we substituted each set of candidate paraphrases into between 2-10 sentences which contained the original phrase.

*** Figure 4 shows the paraphrases for under control substituted into one of the sentences in which it occurred.

*** We created a total of 289 such evaluation sets, with a total of 1366 unique sentences created through substitution.

*** We had two native English speakers produce judgments as to whether the new sentencespreserved the meaning of the original phrase and as to whether they remainedgrammatical.

*** Paraphrases that were judged to preserve both meaning and grammaticality were considered to be correct, and examples which failed on either judgment were considered to be incorrect.

Agent-based sentence

Adjacent sentences

Summary sentence

*** (Bannard and Callison-Burch 2005) replaced phrases with paraphrases in a number of sentences and askedjudges whether the substitutions “preservedmeaningandremainedgrammatical.”

specific content summarization scsum3
Specific Content Summarization (SCSum)
  • Context modeling
    • Choose nearby sentences within a contextual window (size 5) after the agent-based sentence to represent more for given topic.
specific content summarization scsum4
Specific Content Summarization (SCSum)
  • Weighting
    • The observation is that the presence of one or more of current, ancestor and sibling nodes may affect the final score from the computation
    • Add a new weighting coefficient for the score computed from the topic relevance computation (SCSum)

Values as follows:

If sentence contains no keywords in siblings:

+ Keywords in both ancestors & itself  1

+ Keywords in itself only  0.5

+ Keywords in ancestors only  0.25

If sentence contains keywords in siblings  0.1 (penalty)

a weighting coefficient that takes on differing values based on the presence of keywords in the sentence

specific content summarization scsum5
Specific Content Summarization (SCSum)
  • Ranking & Re-ranking
    • Sentences are ranked descendingly according to their relevance scores
    • Then, simplified MMR (SimRank) is performed:
      • A sentence X is removed if it has the maximum cosine similarity value exceeding a pre-defined threshold (0.75) with any sentence Y which is already chosen at previous steps of SimRank.
post proccessing
Post-Proccessing
  • Two steps:
    • First, replace agentive forms (e.g., “we”, “our”, “this study”, ...) with a citation to the articles
      • topic transition using Type 2 together with P1, P2, and C1 patterns for representing mentions to citations
    • Second, resolves abbreviations found in the extracted sentences
      • E.g. SMT  Statistical Machine Translation
generation
Generation
  • In this work, we only generate the related work summaries by using
    • depth-first traversals to form the ordering of topic nodes in a topic tree

Node ordering

1 − 4 −2 − 3 − 5 − 6 − 7

experiments results
Experiments & Results
  • Dataset
    • Use RWSData for evaluation, including 20 sets
      • 10 out of 20 sets were evaluated automatically and manually.
  • Baselines
    • LEAD (title + abstract – based RW)
    • MEAD (centroid + cosine similarity): topic-based summarization
  • Proposed systems
    • ReWoS-WCM (ReWoS without context modeling)
    • ReWoS-CM (ReWoS with context modeling)
experiments results1
Experiments & Results
  • Automatic evaluation
    • Use ROUGE variants (ROUGE-1, ROUGE-2, ROUGE-S4, ROUGE-SU4)
  • Manual evaluation (measure over 5-point scale of 1 (very poor) to 5 (very good)
    • Correctness: Is the summary content actually relevant to the hierarchical topics given?
    • Novelty: Does the summary introduce novel information that is significant in comparison with the human created summary?
    • Fluency: Does the summary’s exposition flow well, in terms of syntax as well as discourse?
    • Usefulness: Is the summary acceptable in terms of its usefulness in supporting the researchers to quickly grasp the related works relevant to hierarchical topics given?
  • Summary length: 1% of the original relevant articles, measured in sentences
experiments results2
Experiments & Results
  • ROUGE evaluation seems to work unreasonably when dealing with verbose summaries, often produced by MEAD.
  • Related work summaries are multi-topic summaries of multi-article references. This may cause miscalculation from overlapping n-grams that occur across multiple topics or references.
experiments results3
Experiments & Results
  • The table shows that both ReWoS–WCM and ReWoS-CM perform significantly better than baseline in terms of correctness, novelty, and usefulness.
  • Comparing with LEAD, showing that necessary information is not only located in titles or abstracts, but also in relevant portions of the research article body.
  • ReWoS–CM (with context modeling) performed equivalent to ReWoS–WCM (without it) in terms of correctness and usefulness.
  • - For novelty, ReWoS–CM is better than ReWoS–WCM. It proved that the proposed component of context moding is useful in providing new information.
future work
Future work
  • An expected fully automated related work summarization

Future directions

Current focus

future work1
Future work
  • Within ReWoS:
    • Context modeling
      • Fusion of contextual sentences
    • More in-dept related work representation
      • Type 1 topic transition with other patterns
  • Break down the assumption of topic hierarchy tree and a set of relevant papers in the input
    • Automating Topic Understanding and Paper Retrieval Components
future works
Future works
  • A feasible algorithm for automatic decomposition of related work summaries
  • A robust automatic evaluation for related work summarization task
  • Go towards practical applications that benefit from automated related work summarization research
conclusion
Conclusion
  • Three main contributions of this work:
    • Constructed a new data set (namely RWSData) specific to the task of related work summarization
    • Conducted a deep manual analysis on various aspects of related work summaries including:
      • Characteristics of RW summaries covered (definition, position, and topical structure)
      • The decomposition and alignment of RW summaries, RW representation, revisions, evaluation metrics.
    • Developed my initial prototype Related Work Summarization system, namely ReWoS:
      • Heuristics-based system
      • Utilizes structure of topic hierarchy tree
      • Implements novel strategies using both general and specific content summarization