TREC 2009 Review

1. TREC 2009 Review Lanbo Zhang

2. 7 tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

3. 67 Participating Groups

4. The new dataset: ClueWeb09 • 1 billion web pages, in 10 languages, half are in English • Crawled by CMU in Jan. and Feb. 2009 • 5 TB (compressed), 25 TB (uncompressed) • Subset B • 50 million English pages • Includes all Wikipedia pages • The original dataset and the Indri index of subset B are available on our lab machines

5. Tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

6. Web Track • Two tasks • Adhoc Retrieval Task • Diversity Task • Return a ranked list of pages that together provide complete coverage for a query, while avoiding excessive redundancy in the return list.

7. Web Track • Topic type 1: ambiguous

8. Web Track • Topic type 2: faceted

9. Web Track • Results of adhoc task

10. Web Track • Results of diversity task

11. Waterloo at Web track • Two runs • Top 10000 docs in the entire collection • Top 10000 docs in the Wikipedia set • Wikipedia docs as pseudo relevance feedback • Machine learning methods to re-rank the top 20000 docs, and return the top 1000 • Diversity task • A Naïve Bayes classifier designed to re-rank the top 20000 to exclude duplicates

12. MSRA at Web track • Mining subtopics for a query by • Anchor texts • Search results clusters • Sites of search results • Search results diversification • A greedy algorithm to iteratively select the next best document

13. Tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

14. Relevance Feedback Track • Tasks • Phase 1: find a set of 5 documents that are good for relevance feedback. • Phase 2: develop an RF algorithm to do retrieval based on the relevance judgments of 5 docs.

15. Results of RF track: Phase 1

16. Results of RF track: Phase 2

17. UCSC at RF track • Phase 1: documents selection • Clustering top ranked documents • Transductive Experimental Design (TED) • Phase 2: RF algorithm • Combining different document representations • Title, anchor, heading, document • Incorporating term position information • Phrase match, text window match • Incorporating document similarities to labeled docs

18. UMas at RF track • A supervised method to estimate the weights of expanded terms for RF • Train collection: wt10g • Term features given a query: • Term frequency in FB docs and entire collection • Co-occurrence with query terms • Term proximity to query terms • Document frequency

19. UMas at RF track • Model: Boosting

20. Tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

21. Entity Track • Task • Given an input entity, find the related entities • Return 100 related entities and their homepages

22. Results of Entity track

23. Purdue at Entity track • Entity Extraction • Hierarchical Relevance Model • Three levels of relevance: document, passage, entity

24. Purdue at Entity track • Homepage Finding for Entities • Logistic Regression model

25. Tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

26. Blog Track • Tasks • Faceted Blog Distillation • Top Stories Identification • Collection: Blogs08 • Crawled between 01/14/2008 and 02/10/2009 • 1.3 million unique blogs

27. Blog Track • Task 1: Faceted Blog Distillation • Given a topic and the faceted restriction, find the relevant blogs. • Facets • Opinionated vs. Factual • Personal vs. Official • In-depth vs. Shallow • Topic example

28. Blog Track • Task 2: Top Stories Identification • Given a date, find the hottest news headlines for that day and select the relevant and diverse blog posts for those headlines • News headlines from New York Times used • Topic example

29. Results of Blog track • Faceted Blog Distillation

30. Results of Blog track • Top Stories Identification • Find the hottest news headlines • Identify the related blog posts

31. BUPT at Blog track • Faceted Blog Distillation • Scoring function: • The title section of a topic plus automatically selected terms from the DESC and NARR sections • Phrase match • Facets Analysis • Opinionated v.s. Factual: a sentiment analysis model • Personal v.s Official: the maximum frequency of an organization entity occurring in a blog (Stanford Named Entity Recognizer) • In-depth v.s. Shallow: post length • Linear combination of the above two parts

32. Univ. of Glasgow at Blog track • Top Stories Identification • The model: • Incorporating the following days • Using Wikipedia to enrich news headline terms and keep the top 10 terms for each headline

33. Tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

34. Legal Track • Tasks • Interactive task (Enron email collection) • Retrieval with topic authorities involved, participants can ask topic authorities to clarify topics, judge the relevance of sample docs • Batch task (IIT CDIP 1.0) • Retrieval with relevance evidence (RF)

35. Results of Legal track

36. Waterloo at Legal track • Interactive task • Phase 1: interactive search and judging • To find a large and diverse set of training examples • Phase 2: interactive learning • To find more potentially relevant documents • Batch task • Run three spam filters on every document: • An on-line logistic regression filter, • A Naïve Bayes spam filter • An on-line version of BM25 RF method

37. Tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

38. Million Query Track • Tasks • Adhoc retrieval for 40000 queries • Predict query types • Query intent: Precision-oriented vs. Recall-oriented • Query difficulty: Hard vs. Easy • Precision-oriented • Navigational: Find a specific URL or web page. • Closed: Find a short, unambiguous answer to a specific question. • Resource: Locate a web-based resource or download. • Recall-oriented • Open: Answer an open-ended question, or nd all available information about a topic. • Advice : Find advice or ideas regarding a general question or problem. • List: Find a list of results that will help satisfy an open-ended goal.

39. Results of Million Query track Precision vs. Recall Hard vs. Easy

40. Northeastern Univ. at MQ track • Query-specific learning to rank • Learn different ranking functions for queries in different classes • Using SVM to classify queries • Training data: MQ 2008 dataset • Features • Document features: document length, TF, IDF, TF*IDF, normalized TF, Robertson’s TF, Robertson’s IDF, BM25, Language Models (Laplace, Dirichlet, JM). • Field features: title, heading, anchor text, and URL • Web graph features

41. Tracks • Web track • Relevance Feedback track (RF) • Entity track • Blog track • Legal track • Million Query track (MQ) • Chemical IR track

42. Chemical IR Track • Tasks • Technical Survey Task • Retrieve documents in response to each topic given by chemical patent experts • Prior Art Search Task • Find relevant patents with respect to a set of 1000 existing patents

43. Results of Chemical track

44. Geneva at Chemical track • Document Representation: • Title,Description,Abstract, Claims, Applicants, Inventors, IPC codes, Patent references • Exploiting Citation Networks • Query expansion using chemical annotations • Filtering based on IPC codes • Re-ranking based on claims