Threshold Setting and Performance Monitoring for Novel Text Mining
Download
1 / 22

- PowerPoint PPT Presentation


  • 174 Views
  • Uploaded on

Threshold Setting and Performance Monitoring for Novel Text Mining. Wenyin Tang and Flora S. Tsai School of Electrical and Electronic Engineering Nanyang Technological University E-mail: [email protected] , [email protected] May 2, 2009. Outline. Introduction

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - presley


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Threshold Setting and Performance Monitoring for Novel Text Mining

Wenyin Tang and Flora S. Tsai

School of Electrical and Electronic EngineeringNanyang Technological University

E-mail: [email protected], [email protected]

May 2, 2009


Outline l.jpg
Outline Mining

  • Introduction

    • Novel Text Mining (NTM) System

    • Performance Evaluation of NTM

  • Adaptive Threshold Setting for NTM

    • Motivations

    • Our Method: Gaussian-based Adaptive Threshold Setting (GATS)

    • Experimental Result

  • Conclusion


Overview of novel text mining system l.jpg
Overview of Novel Text Mining System Mining

Interact with users: input documents, output novel info, preference setting and feedback.

Prepare a clean data matrix which can be easily processed by

a computer.

Categorise each incoming document or sentence into its relevant topic bin.

Detect novel yet relevant documents or sentences in each topic.


Novel text mining algorithm l.jpg
Novel Text Mining Algorithm Mining

Unfortunately, I am

“non-novel” because I am very similar to my nearest neighbor D3

D4

D3

I am “novel” because I am dissimilar with my nearest neighbor D2

D1, D2, D3, D4 …

D2

D1

I am “novel” because I am dissimilar to D1

I am “novel” because I am the first document

Vector space

Given a set of relevant documents in a specific topic, e.g. “football games”, NTM retrieves the novel documents by:

  • Step 1: rank documents in the topic “football games” in a chronological order.

  • Step 2: assign a novelty score for each document by comparing the document with its history documents.

  • Step 3: predict the document as “novel” if its novelty score is greater than the predefined novelty threshold.


Ntm performance evaluation l.jpg
NTM Performance Evaluation Mining

# Novel:

D1, D2, D3, D4, D5, D6, D7, D8, D9, D10

System (S):

8

Assessor (A):

5

Matched (M):

4

  • Precision (P) reflects how likely the system retrieved docs are truly novel. P=M/S=4/8=0.5, i.e. 50% system retrieved docs are truly novel.

  • Recall (R) reflects how likely the truly novel docs can be retrieved by the system. R=M/A=4/5=0.8, i.e. 80% truly novel docs can be retrieved by the system.

  • Fβ score: the function of P and R:

novel

non-novel

  • Given a set of documents D1, D2, to D10, relevant to some topic, for example,


Threshold setting vs users requirements l.jpg
Threshold Setting vs. Users’ Requirements Mining

I want to read the most novel information in a short time1.

I do not want to miss any novel information2.

The NTM system should define the novelty threshold based on the users’ requirements

adaptively.

I am not sure until I can see the documents

  • Different users may have different performance requirements.

  • High-precision NTM systems are desired;

  • High-recall NTM systems are desired.


Why adaptive threshold setting l.jpg
Why Adaptive Threshold Setting Mining

Motivations:

  • As NTM system is a real-time system, there is little or no training information in the initial stages of NTM. Therefore, the threshold cannot be predefined with confidence.

  • As NTM system is an accumulating system, more training information will be available for threshold setting, based on user’s feedback given over time.

  • Different users may have different definitions of “novelty”:

    • One user: a document with 50% novel info

    • Another user: a document with 90% novel info


Gaussian based adaptive threshold setting gats l.jpg
Gaussian-based Adaptive Threshold Setting (GATS) Mining

Basic idea:

  • GATS is a score distribution-based threshold setting method. It models the score distributions of both novel and non-novel documents (based on the user feedback);

  • This parametric model provides the global information of data, from which we can construct an optimization criterion of desired performance to search the best threshold.


Novelty score distributions l.jpg
Novelty Score Distributions Mining

Gaussian probability distribution approximation

Non-novel

Novel

Empirical probability distribution and its Gaussian probability distribution approximation for TREC 2004 Novelty Track data topic N54


Optimization criterion l.jpg
Optimization Criterion Mining

Satisfy 2 conditions:

  • Criterion is a function of Threshold:

    J=f (θ)

    2. Criterion is directly related to system performance:

    J=Fβ(θ)


Optimization criterion11 l.jpg
Optimization Criterion Mining

Non-novel

Novel

S1

S0

θ

θ



Experimental data l.jpg
Experimental Data Mining

Sentence-level data: TREC 2004 Novelty Track data

The news providers of the document set are Xinghua English (XIE) , New York Times (NYT), and Associated Press Worldstream (APW). The NIST assessors created 50 topics for this data. Each topic consists of around 25 documents. These documents were ordered chronologically and then segmented into sentences. Each sentence was given an identifier and concatenated together to form the target sentence set. In this data, the overall percentage of novel sentences is around 41.4%. The statistics of data is summarized in Table 1.

Table 1 Statistics of TREC 2004 Novelty Track data


Experimental data14 l.jpg
Experimental Data Mining

Document-level data: APWSJ

APWSJ consists of news articles from Associate Press (AP) and Wall Street Journal (WSJ), which cover the same period from 1988 to 1990 [Zhang et al., 2002]. There are 50 TREC topics from Q101 to Q150 in this data and 5 topics (Q131, Q142, Q145, Q147, Q150) that lack non-novel documents are excluded from the experiments. The statistics of this data are summarized in Table 2.

Table 2 Statistics of APWSJ data


Methods parameters l.jpg
Methods & Parameters Mining

  • Baseline:

    • Fixed threshold setting θ from 0.05~0.95 with an equal step 0.05.

  • Our method, GATS:

    • Complete feedback: with β from 0.1~0.9 with an equal step 0.1.

    • Partial feedback: with β from 0.1~0.9 with an equal step 0.1, percentages of feedback: 10%, 20%, 50% and 80%.


Experimental result l.jpg
Experimental Result Mining

Sentence-Level NTM on TREC 2004 Data

Precision

Recall


Experimental result17 l.jpg
Experimental Result Mining

Document-Level NTM on APWSJ Data

Redundancy-Precision

Redundancy-Recall


Comparison gats vs fixed threshold l.jpg
Comparison: GATS vs. Fixed Threshold Mining

  • For precision-recall tradeoff

    • Fixed threshold θ cannot reflect the tradeoff of the precision and recall directly.

    • GATS parameter β reflects the weights ofprecision and recall directly.

  • Under various performance requirements, GATS is able to approximate the best fixed threshold.

Table 3 Comparison of Fβ on TREC 2004 Novelty Track data


Experimental result19 l.jpg
Experimental Result Mining

Sentence-Level NTM on TREC 2004 Data

Precision

Recall

PR curves of GATS (tuned for Fβ) with different percentages of the user’s feedback.


Experimental result20 l.jpg
Experimental Result Mining

Document-Level NTM on APWSJ Data

Redundancy-Precision

Redundancy-Recall

R-PR curves of GATS with different percentages of the user’s feedback.


Conclusion l.jpg
Conclusion Mining

A Gaussian-based Adaptive Threshold Setting (GATS) algorithm was proposed for NTM system.

GATS is a generic method, which can be tuned according to different performance requirements varying from high-precision to high-recall.

By testing the proposed method on both document and sentence-level datasets, we found the experimental results showed the promising performance of GATS for a real-time NTM system.


Slide22 l.jpg

Thank You! Mining

Q & A


ad