1 / 96

Information Retrieval and Search Engines

Information Retrieval and Search Engines. Lecture 7: Probabilistic Information Retrieval and Language Models Prof . Michael R. Lyu. Motivation of This Lecture. Given an information need and a collection of documents , a system must determine how well the documents satisfy the query

Download Presentation

Information Retrieval and Search Engines

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Information Retrieval and Search Engines Lecture 7: Probabilistic Information Retrieval and Language Models Prof. Michael R. Lyu

  2. Motivation of This Lecture • Given an information need and a collection of documents, a system must determine how well the documents satisfy the query • An IR system has an uncertain understanding of the user query, and makes an uncertain guess of whether a document satisfies the query • Probability theory provides a principled foundation for such reasoning under uncertainty • Probabilistic models exploit this foundation to estimate how likely it is that a document is relevant to a query

  3. Motivation of This Lecture A document is generated from a language model Estimate parameters of language model Measure the probability of a document generates a query generate document

  4. Outline (Ch. 11, 12 of IR Book) • Recap • Probabilistic Information Retrieval • Probabilistic Approach to Retrieval • Basic Probability Theory • Probability Ranking Principle • Appraisal & Extensions • Language Models for Information Retrieval • Language Models • Query Likelihood Model • Language Model versus Other Approaches • Extended Language Modeling Approaches

  5. Outline (Ch. 11, 12 of IR Book) • Recap • Probabilistic Information Retrieval • Probabilistic Approach to Retrieval • Basic Probability Theory • Probability Ranking Principle • Appraisal & Extensions • Language Models for Information Retrieval • Language Models • Query Likelihood Model • Language Model versus Other Approaches • Extended Language Modeling Approaches

  6. Evaluation in IR • User happiness can only be measured by relevance to an information need, not by relevance to queries

  7. A Combined Measure: F • F allows us to trade off precision against recall • where • α ϵ[0, 1] and thus b2ϵ [0,∞] • Most frequently used: balanced F (F1)with b = 1 or α = 0.5 • This is the harmonic mean of P and R: • Value of b < 1 emphasize precision • Value of b > 1 emphasize recall 7

  8. RelevanceFeedback: Basic Idea • Basic Idea • It may be difficult to formulate a good query when you don’t know the collection well, or cannot express it, but can judge relevance of a result. So iterate … • The user issues a (short, simple) query • The search engine returns a set of documents • User marks some docs as relevant, some as non-relevant • Search engine computes a new representation of the information need. Hope: better than the initial query • Search engine runs new query and returns new results • New results have (hopefully) better recall

  9. RelevanceFeedback: Example 1

  10. ResultsforInitial Query

  11. User Feedback: Select Whatis Relevant

  12. Results after RelevanceFeedback

  13. RocchioAlgorithm (SMART) • qm: modified query vector; q0: original query vector • Dr: sets of known relevant • Dnr : sets of non-relevant documents respectively • α, β, and γ: weights • New query moves towards relevant documents and away from non-relevant documents

  14. Types of Query Expansion • Manual thesaurus • Controlled vocabulary. • E.g. MedLine: physician, synonym: doc, doctor, MD, medico • Can be query rather than just synonyms • Global Analysis: (static; of all documents in collection) • Automatically derived thesaurus • (co-occurrence statistics) • Refinements based on query log mining • Common on the web • Local Analysis: (dynamic) • Analysis of documents in result set

  15. Outline (Ch. 11, 12 of IR Book) • Recap • Probabilistic Information Retrieval • Probabilistic Approach to Retrieval • Basic Probability Theory • Probability Ranking Principle • Appraisal & Extensions • Language Models for Information Retrieval • Language Models • Query Likelihood Model • Language Model versus Other Approaches • Extended Language Modeling Approaches

  16. Probabilistic Model VS. Other Models • Boolean model • Boolean model does not support ranking • Vector space model • A formally defined model that supports ranking • Rank documents according to similarity to query • The notion of similarity does not translate directly into an assessment of “is the document a good document to give to the user or not?” • Probability theory is arguably a cleaner formalization of what we really want an IR system to do: give relevant documents to the user

  17. Probabilistic Approach to Retrieval • An IR system has an uncertain understanding of the user query , and makes an uncertain guess of whether a document satisfies the query • Probability theory provides a principled foundation for such reasoning under uncertainty • Probabilistic models exploit this foundation to estimate how likely it is that a document is relevant to a query

  18. Probabilistic IR Models at a Glance • Classical probabilistic retrieval model • Probability ranking principle • Binary Independence Model • BestMatch25 (Okapi) • Bayesian networks for text retrieval • Language model approach to IR • Probabilistic methods are one of the oldest but also one of the currently hottest topics in IR

  19. Outline (Ch. 11, 12 of IR Book) • Recap • Probabilistic Information Retrieval • Probabilistic Approach to Retrieval • Basic Probability Theory • Probability Ranking Principle • Appraisal & Extensions • Language Models for Information Retrieval • Language Models • Query Likelihood Model • Language Model versus Other Approaches • Extended Language Modeling Approaches

  20. Basic Probability Theory • For events A and B • Joint probability P(A, B), both events occurring • Conditional probability P(A|B) of event A occurring given that event B has occurred. • Chain rule gives fundamental relationship between joint and conditional probabilities

  21. Basic Probability Theory • Similarly for the complement of an event • Partition rule: if B can be divided into an exhaustive set of disjoint subcases, then P(B) is the sum of the probabilities of the subcases • A special case of this rule gives: A B

  22. Basic Probability Theory • Bayes’ Rulefor inverting conditional probabilities • Can be thought of as a way of updating probabilities • Start off with prior probability P(A) (initial estimate of how likely event A is in the absence of any other information) • Derive a posterior probability P(A|B) after having seen the evidence B, based on the likelihood of B occurring in the two cases that A does or does not hold

  23. Basic Probability Theory • Odds of an event provide a kind of multiplier for how probabilities change:

  24. Outline (Ch. 11, 12 of IR Book) • Recap • Probabilistic Information Retrieval • Probabilistic Approach to Retrieval • Basic Probability Theory • Probability Ranking Principle • Appraisal & Extensions • Language Models for Information Retrieval • Language Models • Query Likelihood Model • Language Model versus Other Approaches • Extended Language Modeling Approaches

  25. The Document Ranking Problem • Ranked retrieval setup • Given a collection of documents, the user issues a query, and an ordered list of documents is returned • Assume binary notion of relevance • Rd,qis a random dichotomous variable • Rd,q= 1, if document d is relevant w.r.t query q • Rd,q = 0, otherwise • Probabilistic ranking orders documents decreasingly by their estimated probability of relevance w.r.t. query: P(R = 1|d, q)

  26. Probability Ranking Principle (PRP) • PRP in brief • If the retrieved documents (w.r.t a query) are ranked decreasingly on their probability of relevance, then the effectiveness of the system will be the best that is obtainable • PRP in full • If [the IR] system’s response to each [query] is a ranking of the documents in order of decreasing probability of relevance to the [query], where the probabilities are estimated as accurately as possible on the basis of whatever data have been made available to the system for this purpose, the overall effectiveness of the system to its user will be the best that is obtainable on the basis of those data

  27. Binary Independence Model (BIM) • Traditionally used with the PRP • Binary (equivalent to Boolean): documents andqueries represented as binarytermincidencevectors • E.g., document d represented by vector x = (x1, . . . , xM), where xt = 1 if term t occurs in d and xt = 0 otherwise • Different documents may have the same vector representation • Independence: no association between terms (not true, but practically works - ‘naive’ assumption of Naive Bayes models)

  28. Binary Incidence Matrix Each document is represented as a binary vector

  29. Binary Independence Model • To make a probabilistic retrieval strategy precise, need to estimate how terms in documents contribute to relevance • Find measurable statistics (term frequency, document frequency, document length) that affect judgments about documentrelevance • Combine these statistics to estimate the probability of documentrelevance • Order documents by decreasing estimated probability of relevanceP(R|d,q) • Assume that the relevance of each document is independent of the relevance of other documents (not true, in practice allowsduplicateresults)

  30. Binary Independence Model • is modeled using term incidence vectors as • and : • Probability that if a relevant or nonrelevantdocument is retrieved, then that document’srepresentationis • Statistics about the actual document collection are used to estimatetheseprobabilities Bayes’ Theorem

  31. Binary Independence Model • and : prior probability of retrieving a relevant or non-relevant document for a query q • Estimate and from percentage of relevant documents in thecollection • Since a document is either relevant or non-relevant to a query, we must havethat: Same equation as on previous slide

  32. Deriving a Ranking Function for Query Terms (1) • Given a query q, ranking documents by is modeled under BIM as ranking them by • Easier: rank documents by their odds of relevance (gives same ranking & we can ignore the common denominator) • is a constant for a given query - can be ignored Drop this term

  33. Deriving a Ranking Function for Query Terms (2) • We make the Naive Bayes conditional independence assumptionthat the presence or absence of a word in a document is independent of the presence or absence of any other word (given the query) • So: All terms in the dictionary Drop this term

  34. Deriving a Ranking Function for Query Terms (3) • Since each xt is either 0 or 1, we can separate the terms to give: • Let be the probability of a term appearing in relevant document • Let be the probability of a term appearing in a non-relevant document • Visualise as contingency table Drop this term

  35. Deriving a Ranking Function for Query Terms (4) • Additional simplifying assumption • Terms not occurring in the query are equally likely to occur in relevant and non-relevantdocuments • If qt = 0, then pt = ut • We need only considerterms in the products that appear in the query

  36. Deriving a Ranking Function for Query Terms (5) • Including the query terms found in the document into the right product, but simultaneously dividing through by them in the left product • The left product is still over query terms found in the document, but the right product is now over all query terms, hence constant for a particular query and can be ignored • The only quantity that needs to be estimated to rank documents w.r.t a query is the left product Drop this term

  37. Retrieval Status Value (RSV) • Equally rank documents by the logarithm of this term, since log is a monotonic function • Retrieval status value (RSV) • Resulting quantity used for ranking

  38. Log Odds Ratio • Everything comes down to computing the RSV • We can equally rank documents using the log odds ratiosfor the terms in the query ct • The odds ratio is the ratio of two odds • The odds ofthe term appearing ifthe document is relevant (pt/(1 − pt)) • The odds of the termappearing if the documentisnon-relevant (ut/(1 − ut))

  39. What ct Means? • ct = 0: a term has equal odds of appearing in relevant and non-relevant documents • ctpositive: it is more likely to appear in relevantdocuments • ctnegative: it is more likely to appear in non-relevantdocuments • ct functions as a term weight • Operationally, we sum ct quantities in accumulators for query terms appearing in documents, just as for the vector space model calculations

  40. Calculate pt, ut and ct • For each term t in a query, estimate ctin the whole collection using a contingency table of counts of documents in the collection, where dft is the number of documents that contain term t Term present in relevant document Term present in non-relevant document

  41. Exercise • Suppose a collection have 100 documents, a user has an information need Q, and the user uses a single word q to represent his/her information need. There are 40 documents relevant to the information need Q. Among the 40 relevant documents, only 30 documents contain the word q. Among the whole collection, 80 documents contain the word q. • Calculate pt, ut and ct for above scenario.

  42. Answer to Exercise Pt = 30/40 = ¾ Ut = 50/60 = 5/6 Ct = log[(3/4)/(1/4)] – log[(5/6)/(1/6)]=log 3 – log 5

  43. Expected Likelihood Estimation • To avoid the possibility of zeroes (such as if every or no relevant document has a particular term), apply smoothing • Add 0.5 to each count (Expected likelihood estimation, ELE)

  44. Maximum A Posteriori • One simple way of smoothing is to add a number αto each of the observedcounts • These pseudocounts correspond to the use of a uniformdistribution over the vocabulary as a Bayesianprior • The value of α denotes the strength of our belief in uniformity • Maximum A Posteriori (MAP) estimation • Choose the most likely point value for probabilities based on the prior and the observedevidence

  45. In Class Practice 1 • Link

  46. Probability Estimates in Practice • Assuming that relevant documents are a very smallpercentage of the collection, approximatestatistics for non-relevant documents by statistics from the whole collection • Hence, ut (the probability of term occurrence in non-relevant documents for a query) is dft/N and • log[(1 − ut )/ut ] = log[(N − dft)/df t ] ≈ log N/df t • Provide a theoreticaljustification for the mostfrequentlyusedform of idfweighting • The above approximation cannot easily be extended to relevant documents

  47. Probability Estimate in Practice • Estimate quantity pt • Croft and Harper, Journal of Documentation, 1979 • Using a constant in combination match model • Assume that pt is constant over all terms xt in the query, pt=0.5 • Pt and (1-pt) factors cancel out in RSV. Combining this with earlier approximation for ut. Document ranking is determinedly by which query term occur in documents scaled by their idf weighting • Suitable for short documents

  48. Probability Estimate in Practice • Estimate quantity pt • Grieff, SIGIR, 1998 • Argue that constant estimate of pt in the Croft and Harper (1997) model is theoretically problematic and not observed empirically • A better estimate is simply estimating pt from collection level statistics about the occurrence of t, as pt = dft/N

  49. Outline (Ch. 11, 12 of IR Book) • Recap • Probabilistic Information Retrieval • Probabilistic Approach to Retrieval • Basic Probability Theory • Probability Ranking Principle • Appraisal & Extensions • Language Models for Information Retrieval • Language Models • Query Likelihood Model • Language Model versus Other Approaches • Extended Language Modeling Approaches

  50. History and Summary of Assumptions • Among the oldest formal models in IR • Maron & Kuhns, 1960: Since an IR system cannot predict with certainty which document is relevant, we should deal with probabilities • Assumptions for getting reasonable approximations of the needed probabilities (in the BIM) • Boolean representationofdocuments/queries/relevance • Termindependence • Out-of-querytermsdonotaffect retrieval • Documentrelevancevaluesareindependent

More Related