Evaluating Information Retrieval Methods for Computer Science Research: A Comprehensive Overview

1. Special Topics in Computer ScienceAdvanced Topics in Information RetrievalChapter 3: Goals: Retrieval Evaluation Alexander Gelbukh www.Gelbukh.com

2. Previous chapter • Models are needed for formal operations • Boolean model is the simplest • Vector model is the best combination of quality and simplicity • TF-IDF term weighting • This (or similar) weighting is used in all further models • Many interesting and not well-investigated variations • possible future work

3. Previous chapter: Research issues • How people judge relevance? • ranking strategies • How to combine different sources of evidence? • What interfaces can help users to understand and formulate their information need? • user interfaces: an open issue • Meta-search engines: how to combine results from different Web search engines? • These results almost do not intersect • How to combine rankings?

4. To write a paper: Evaluation! • How do you measure whether a system is good or bad? • To go to the right direction, need to know where you want to get to. • “We can do it this way” vs. “This way it performs better” • “I think it is better...” • “We do it this way...” • “Our method takes into account syntax and semantics...” • “I like the results...” • Criterion of truth. Crucial for any science. • Enables competition  financial policy attracts people • TREC international competitions 

5. Methodology to write a paper • Define formally your task and constraints • Define formally your evaluation criterion (argue if needed) • One numerical value is better than several • Show that your method gives better value than • the baseline (the simple obvious way), such as: • Retrieve all. Retrieve none. Retrieve at random. Use Google. • state-of-the-art (the best reported method) • in the same setting and same evaluation method! • and your parameter settings are optimal • Consider extreme settings: 0, 

6. ... Methodology The only valid way of reasoning • “But we want the clusters to be non-trivial” • Add this as a penalty to your criteria or as constraints • Divide your “acceptability considerations” into: • Constraints: yes/no. • Evaluation: better/worse. • Check that your evaluation criteria are well justified • “My formula gives it this way” • “My result is correct since this is what my algorithm gives” • Reason in terms of the user task, not your algorithm / formulas • Are your good/bad judgments in accord with intuition?

7. Evaluation? (Possible? How?) • IR: “user satisfaction” • Difficult to model formally • Expensive to measure directly (experiments with subjects) • At least two contradicting parameters • Completeness vs. quality • No good way to combine into one single numerical value • Some “user-defined” “weights of importance” of the two • Not formal, depend on situation • Art

8. Parameters to evaluate • Performance (in general sense) • Speed • Space • Tradoff • Common for all systems. Not discussed here. • Retrieval performance (quality?) • = goodness of a retrieval strategy • A testreference collection: docs and queries. • The “correct” set (or ordering) provided by “experts” • A similarity measure to compare system output with the “correct” one.

9. Evaluation: Model User Satisfaction • User task • Batch query processing? Interaction? Mixed? • Way of use • Real-life situation: what factors matter? • Interface type • In this chapter: laboratory settings • Repeatability • Scalability

10. Sets (Boolean): Precision & Recall • Tradeoff (as with time and space) • Assumes the retrieval results are sets • as in Boolean; in Vector, use threshold • Measures closeness between two sets • Recall:Of relevant docs, how many (%) were retrieved? Others are lost. • Precision:Of retrieved docs, how many (%) are relevant? Others are noise. • Nowadays with huge collections Precision is more important!

11. Precision & Recall Recall = Precision =

12. Ranked Output (Vector): ? • “Truth”: ordering built by experts • System output: guessed ordering • Ways to compare two rankings: ? • Build the “truth” set is not possible or too expensive • So not used (rarely used?) in practice • One can built the “truth” set automatically • Research topic for us?

13. Ranked Output (Vector) vs. Set • “Truth”: unordered “relevant” set • Output: ordered guessing • Compare ordered set with an unordered one

14. ... Ranked Output vs. set (one query) • Plot precision vs. recall curve • In the initial part of the list containing n% of all relevant docs, what the precision is? • 11 standard recall levels: 0%, 10%, ..., 90%, 100%. • 0%: interpolated

15. ... Many queries • Average precision and recall Ranked output: • Average precision at each recall level • To get equal (standard) recall levels, interpolation • of 3 relevant docs, there is no 10% level! • Interpolated value at level n =maximum known value between n and n + 1 • If none known, use the nearest known.

16. Precision vs. Recall Figures • Alternative method: document cutoff values • Precision at first 5, 10, 15, 20, 30, 50, 100 docs • Used to compare algorithms. • Simple • Intuitive • NOT a one-value comparison!

17. Which one is better?

18. Single-value summaries • Curves cannot be used for averaging by multiple queries • We need single-value performance for each query • Can be averaged over several queries • Histogram for several queries can be made • Tables can be made • Precision at first relevant doc? • Average precision at (each) seen relevant docs • Favors systems that give several relevant docs first • R-precision • precision at R-th retrieved (R = total relevant)

19. Precision histogram Two algs: A, B R(A)-R(B). Which is better?

20. Alternative measures for Boolean • Problems with Precision & Recall measure: • Recall cannot be estimated with large collections • Two values, but we need one value to compare • Designed for batch mode, not interactive. Informativeness! • Designed for linear ordering of docs (not weak ordering) • Alternative measures: combine both in one F-measure: E-measure:user preference Rec vs. Prec

21. User-oriented measures Definitions:

22. User-oriented measures • Coverage ratio • Many expected docs • Novelty ratio • Many new docs • Relative recall: # found / # expected • Recall effort: # expected / # examined until those are found • Other: • expected search length (good for weak order) • satisfaction (considers only relevant docs) • frustration (considers only non-relevant docs)

23. Reference collections Texts with queries and relevant docs known TREC • Text REtrieval Conference. Different in different years • Wide variety of topics. Document structure marked up. • 6 GB. See NIST website: available at small cost • Not all relevant docs marked! • Pooling method: • top 100 docs in ranking of many search engines • manually verified • Was tested that is a good approximation to the “real” set

24. Ad-hoc (conventional: query  answer) Routing (ranked filtering of changing collection) Chinese ad-hoc Filtering (changing collection; no ranking) Interactive (no ranking) NLP: does it help? Cross-language (ad-hoc) High precision (only 10 docs in answer) Spoken document retrieval (written transcripts) Very large corpus (ad-hoc, 20 GB = 7.5 M docs) Query task (several query versions; does strategy depends on it?) Query transforming Automatic Manual ...TREC tasks

25. ...TREC evaluation • Summary table statistics • # of requests used in the task • # of retrieved docs; # of relevant retrieved and not retrieved • Recall-precision averages • 11 standard points. Interpolated (and not) • Document level averages • Also, can include average R-value • Average precision histogram • By topic. • E.g., difference between R-precision of this system and average of all systems

26. Smaller collections • Simpler to use • Can include info that TREC does not • Can be of specialized type (e.g., include co-citations) • Less sparse, greater overlap between queries • Examples: • CACM • ISI • there are others

27. CACM collection • Communications of ACM, 1958-1979 • 3204 articles • Computer science • Structure info (author, date, citations, ...) • Stems (only title and abstract) • Good for algorithms relying on cross-citations • If a paper cites another one, they are related • If two papers cite the same ones, they are related • 52 queries with Boolean form and answer sets

28. ISI collection • On information sciences • 1460 docs • For similarity in terms and cross-citation • Includes: • Stems (title and abstracts) • Number of cross-citations • 35 natural-language queries with Boolean form and answer sets

29. Cystic Fibrosis (CF) collection • Medical • 1239 docs • MEDLINE data • keywords assigned manually! • 100 requests • 4 judgments for each doc • Good to see agreement • Degrees of relevance, from 0 to 2 • Good answer set overlap • can be used for learning from previous queries

30. Research issues • Different types of interfaces; interactive systems: • What measures to use? • Such as infromativeness

31. Conclusions • Main measures: Precision & Recall. • For sets • Rankings are evaluated through initial subsets • There are measures that combine them into one • Involve user-defined preferences • Many (other) characteristics • An algorithm can be good at some and bad at others • Averages are used, but not always are meaningful • Reference collection exists with known answers to evaluate new algorithms

32. Thank you! Till ... ??