CS276A Information Retrieval

1. CS276AInformation Retrieval Lecture 8

2. Recap of the last lecture • Vector space scoring • Efficiency considerations • Nearest neighbors and approximations

3. This lecture • Results summaries • Evaluating a search engine • Benchmarks • Precision and recall

4. Results summaries

5. Summaries • Having ranked the documents matching a query, we wish to present a results list • Typically, the document title plus a short summary • Title – typically automatically extracted • What about the summaries?

6. Summaries • Two basic kinds: • Static and • Query-dependent (Dynamic) • A static summary of a document is always the same, regardless of the query that hit the doc • Dynamic summaries attempt to explain why the document was retrieved for the query at hand

7. Static summaries • In typical systems, the static summary is a subset of the document • Simplest heuristic: the first 50 (or so – this can be varied) words of the document • Summary cached at indexing time • More sophisticated: extract from each document a set of “key” sentences • Simple NLP heuristics to score each sentence • Summary is made up of top-scoring sentences. • Most sophisticated, seldom used for search results: NLP used to synthesize a summary

8. Dynamic summaries • Present one or more “windows” within the document that contain several of the query terms • Generated in conjunction with scoring • If query found as a phrase, the occurrences of the phrase in the doc • If not, windows within the doc that contain multiple query terms • The summary itself gives the entire content of the window – all terms, not only the query terms – how?

9. Generating dynamic summaries • If we have only a positional index, cannot (easily) reconstruct context surrounding hits • If we cache the documents at index time, can run the window through it, cueing to hits found in the positional index • E.g., positional index says “the query is a phrase in position 4378” so we go to this position in the cached document and stream out the content • Most often, cache a fixed-size prefix of the doc • Cached copy can be outdated

10. Evaluating search engines

11. Measures for a search engine • How fast does it index • Number of documents/hour • (Average document size) • How fast does it search • Latency as a function of index size • Expressiveness of query language • Speed on complex queries

12. Measures for a search engine • All of the preceding criteria are measurable: we can quantify speed/size; we can make expressiveness precise • The key measure: user happiness • What is this? • Speed of response/size of index are factors • But blindingly fast, useless answers won’t make a user happy • Need a way of quantifying user happiness

13. Measuring user happiness • Issue: who is the user we are trying to make happy? • Depends on the setting • Web engine: user finds what they want and return to the engine • Can measure rate of return users • eCommerce site: user finds what they want and make a purchase • Is it the end-user, or the eCommerce site, whose happiness we measure? • Measure time to purchase, or fraction of searchers who become buyers?

14. Measuring user happiness • Enterprise (company/govt/academic): Care about “user productivity” • How much time do my users save when looking for information? • Many other criteria having to do with breadth of access, secure access … more later

15. Happiness: elusive to measure • Commonest proxy: relevance of search results • But how do you measure relevance? • Will detail a methodology here, then examine its issues • Requires 3 elements: • A benchmark document collection • A benchmark suite of queries • A binary assessment of either Relevant or Irrelevant for each query-doc pair

16. Evaluating an IR system • Note: information need is translated into a query • Relevance is assessed relative to the information neednot thequery • E.g., Information need: I'm looking for information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine. • Query: wine red white heart attack effective

17. Standard relevance benchmarks • TREC - National Institute of Standards and Testing (NIST) has run large IR test bed for many years • Reuters and other benchmark doc collections used • “Retrieval tasks” specified • sometimes as queries • Human experts mark, for each query and for each doc, Relevant or Irrelevant • or at least for subset of docs that some system returned for that query

18. Precision and Recall • Precision: fraction of retrieved docs that are relevant = P(relevant|retrieved) • Recall: fraction of relevant docs that are retrieved = P(retrieved|relevant) • Precision P = tp/(tp + fp) • Recall R = tp/(tp + fn)

19. Accuracy • Given a query an engine classifies each doc as “Relevant” or “Irrelevant”. • Accuracy of an engine: the fraction of these classifications that is correct.

20. Why not just use accuracy? • How to build a 99.9999% accurate search engine on a low budget…. • People doing information retrieval want to find something and have a certain tolerance for junk. Snoogle.com Search for: 0 matching results found.

21. Precision/Recall • Can get high recall (but low precision) by retrieving all docs for all queries! • Recall is a non-decreasing function of the number of docs retrieved • Precision usually decreases (in a good system)

22. Difficulties in using precision/recall • Should average over large corpus/query ensembles • Need human relevance assessments • People aren’t reliable assessors • Assessments have to be binary • Nuanced assessments? • Heavily skewed by corpus/authorship • Results may not translate from one domain to another

23. A combined measure: F • Combined measure that assesses this tradeoff is F measure (weighted harmonic mean): • People usually use balanced F1measure • i.e., with  = 1 or  = ½ • Harmonic mean is conservative average • See CJ van Rijsbergen, Information Retrieval

24. F1 and other averages

25. Ranked results • Evaluation of ranked results: • You can return any number of results • By taking various numbers of returned documents (levels of recall), you can produce a precision-recall curve

26. Precision-recall curves

27. Interpolated precision • If you can increase precision by increasing recall, then you should get to count that…

28. Evaluation • There are various other measures • Precision at fixed recall • Perhaps most appropriate for web search: all people want are good matches on the first one or two results pages • 11-point interpolated average precision • The standard measure in the TREC competitions: you take the precision at 11 levels of recall varying from 0 to 1 by tenths of the documents, using interpolation (the value for 0 is always interpolated!), and average them

29. Creating Test Collectionsfor IR Evaluation

30. Test Corpora

31. From corpora to test collections • Still need • Test queries • Relevance assessments • Test queries • Must be germane to docs available • Best designed by domain experts • Random query terms generally not a good idea • Relevance assessments • Human judges, time-consuming • Are human panels perfect?

32. Kappa measure for inter-judge (dis)agreement • Kappa measure • Agreement among judges • Designed for categorical judgments • Corrects for chance agreement • Kappa = [ P(A) – P(E) ] / [ 1 – P(E) ] • P(A) – proportion of time coders agree • P(E) – what agreement would be by chance • Kappa = 0 for chance agreement, 1 for total agreement.

33. P(A)? P(E)? Kappa Measure: Example

34. Kappa Example • P(A) = 370/400 = 0.925 • P(nonrelevant) = (10+20+70+70)/800 = 0.2125 • P(relevant) = (10+20+300+300)/800 = 0.7878 • P(E) = 0.2125^2 + 0.7878^2 = 0.665 • Kappa = (0.925 – 0.665)/(1-0.665) = 0.776 • For >2 judges: average pairwise kappas

35. Kappa Measure • Kappa > 0.8 = good agreement • 0.67 < Kappa < 0.8 -> “tentative conclusions” (Carletta 96) • Depends on purpose of study

36. Interjudge Agreement: TREC 3

37. Impact of Inter-judge Agreement • Impact on absolute performance measure can be significant (0.32 vs 0.39) • Little impact on ranking of different systems or relative performance

38. Unit of Evaluation • We can compute precision, recall, F, and ROC curve for different units. • Possible units • Documents (most common) • Facts (used in some TREC evaluations) • Entities (e.g., car companies) • May produce different results. Why?

39. Critique of pure relevance • Relevance vs Marginal Relevance • A document can be redundant even if it is highly relevant • Duplicates • The same information from different sources • Marginal relevance is a better measure of utility for the user. • Using facts/entities as evaluation units more directly measures true relevance. • But harder to create evaluation set • See Carbonell reference

40. Can we avoid human judgment? • Not really • Makes experimental work hard • Especially on a large scale • In some very specific settings, can use proxies • Example below, approximate vector space retrieval

41. Approximate vector retrieval • Given n document vectors and a query, find the k doc vectors closest to the query. • Exact retrieval – we know of no better way than to compute cosines from the query to every doc • Approximate retrieval schemes – such as cluster pruning in lecture 6 • Given such an approximate retrieval scheme, how do we measure its goodness?

42. Approximate vector retrieval • Let G(q) be the “ground truth” of the actual k closest docs on query q • Let A(q) be the k docs returned by approximate algorithm A on query q • For precision and recall we would measure A(q) G(q) • Is this the right measure?

43. Alternative proposal • Focus instead on how A(q) compares to G(q). • Goodness can be measured here in cosine proximity to q: we sum up qd over d A(q). • Compare this to the sum of qd over d G(q). • Yields a measure of the relative “goodness” of A vis-à-vis G. • Thus A may be 90% “as good as” the ground-truth G, without finding 90% of the docs in G. • For scored retrieval, this may be acceptable: • Most web engines don’t always return the same answers for a given query.

44. Resources for this lecture • MIR Chapter 3 • MG 4.5