CMU TDT Report TIDES PI Meeting 2002

1. CMU TDT Report TIDES PI Meeting 2002 The CMU TDT Team: Jaime Carbonell, Yiming Yang, Ralf Brown, Jian Zhang, Nianli Ma, Chun Jin Language Technologies Institute, CMU

2. Time Line for TDT Activities • ReStarted TDT: Summer 2001 • Tasks: FSD, SLD, Detection • New Techniques: Nov 2001 – Present • Topic-conditional Novelty (FSD) • Situated NE’s (all tasks) • Source-conditional interpolated training (SLD) • Evaluations • TDT: Oct 2001, July 2002 • New FSD (internal): July 2002 (KDD Conference)

3. 2002 Dry Run Results: DET [1] Using our Mandarin to English EBMT, and replace our boundary with systran’s boundary. [2] Using our Dictionary-Based Arabic to English translation, and with our own boundaries. So the boundaries of evaluation and our results are mismatching. [3] Using our Dictionary-Based Arabic to English translation, and replace our boundary with systran’s boundary.

4. Baseline FSD Method • (Unconditional) Dissimilarity with Past • Decision threshold on most-similar story • (Linear) temporal decay • Length-filter (for teasers) • Cosine similarity with standard weights:

5. 2002 Dry Run Results: FSD

6. 2002 Dry Run DET: CMU-FSD

7. FSD Observations • Cross-site comparable baselines (cost =.7) • “Events-vs-Topics” issue (e.g. Asia crisis) • A few mislabled stories wreak havoc for FSD • Eager auto-segmentation a problem (misses) • Recommendations for TDT labeling • FSD on true events, or events within topic(s) • Change auto-segmentation optimality criterion ?? • Recommendations for TDT reserachers • Keep working hard on FSD – not cracked yet

8. New FSD Directions • Topic-conditional models • E.g. “airplane,” “investigation,” “FAA,” “FBI,” “casualties,”  topic, not event • “TWA 800,” “March 12, 1997”  event • First categorize into topic, then use maximally-discriminative terms within topic • Rely on situated named entities • E.g. “Arcan as victim,” “Sharon as peacemaker”

9. Broad Topics vs Events

10. Two-level Scheme for FSD

11. Confusability between Intra-topic Events • AIRPLANE ACCIDENTS BOMBINGS • Each data point in the matrix is the similarity between the two corresponding documents. • Documents are sorted by event as the first key and by the time of arrival as second key, so the diagonal sub-matrices are intra-event document similarities, while the off-diagonal sub-matrices are inter-event document similarities.

12. Measuring Effectiveness of NEs [1] f means a Named Entity; Sk the Kth type of Named Entities among seven types of NEs. [2] We use the effectiveness of each type of NEs to measure how well they can differentiate intra-topic events.

13. Effectiveness of Named Entities

14. Experimental Design • Baseline: conventional FSD • Simple case: two-level FSD with “perfect” topic labels • Ideal case: two-level FSD with “perfect” topic labels, weighted NE and removing topic-specific stop words • Real case: the same as Ideal Case except using system-predicted topic labels

15. Data Description • Broadcast News: published by Primary Source Media, • 261,209 transcripts for news articles from ABC, CNN, NPR and MSNBC in the period from 1992 to 1998. • Document Structure: each document (story) is composed of several fields, such as Title, Topic, Keywords, Date, Abstract and Body. • (Training) topic labels provided by PSM (4 topics) • Airplane accidents, bombings, tornados, hijackings • CMU students labeled 36 events within 4 topics (divided into 50% training and 50% test)

16. Results for Topic-Conditioned FSD

17. Confusability Reduction (5 events within topic: airplane accident in test data) NOTE: • These graphs only contains test data (5 events for topic “airplane accidents”) • The left graph is the Baseline, and the right one is the Ideal Case.

18. Topic-Conditioned Approach to First Story Detection for TDT