1 / 23

Language Models for Information Retrieval

Language Models for Information Retrieval. Andy Luong and Nikita Sudan. Outline. Language Model Types of Language Models Query Likelihood Model Smoothing Evaluation Comparison with other approaches. Language Model.

zorita-craig
Download Presentation

Language Models for Information Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Language Models for Information Retrieval Andy Luong and Nikita Sudan

  2. Outline • Language Model • Types of Language Models • Query Likelihood Model • Smoothing • Evaluation • Comparison with other approaches

  3. Language Model • A language model is a function that puts a probability measure over strings drawn from some vocabulary.

  4. Language Models P(q|Md) instead of P(R=1|q,d)

  5. Example • Doc1: “frog said that toad likes frog” • Doc2: “toad likes frog” 1/3 1/6 1/3

  6. Example Continued q = “frog likes toad” P(q | M1) = (1/3)*(1/6)*(1/6)*0.8*0.8*0.2 P(q | M2) = (1/3)*(1/3)*(1/3)*0.8*0.8*0.2 P(q | M1) < P (S | M2)

  7. Types of Language Models CHAIN RULE UNIGRAM LM BIGRAM LM

  8. Multinomial distribution Frequency Order Constraint M is the size of the term vocabulary

  9. Query Likelihood Model ≈

  10. Query Likelihood Model • Infer LM for each document • Estimate P(q | Md(i)) • Rank documents based on probabilities

  11. MLE

  12. Smoothing • Basic Intuition • New word or unseen word in the document • P( t | Md) = 0 • Zero probabilities will make P ( q | Md) = 0 • Why else should we smooth?

  13. Smoothing Continued Non-occurring term Probability Bound Linear Interpolation Language Model

  14. Example • Doc1: “frog said that toad likes frog” • Doc2: “toad likes frog” 1/3 1/9 1/9 2/9 2/9

  15. Example Continued q= “frog said” λ = ½ P(q | M1) = [(1/3 + 1/3)*(1/2)] * [(1/6 + 1/9)*(1/2)] = .046 P(q | M2) = [(1/3 + 1/3)*(1/2)] * [(0 + 1/9)*(1/2)] = .018 P(q | M1) > P (q | M2)

  16. Evaluation • Precision = (relevant documents ∩ retrieved documents)/ retrieved documents • Recall = (relevant documents ∩ retrieved documents)/ relevant documents

  17. Tf-Idf • The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.

  18. Ponte and Croft’s Experiments

  19. Pros and Cons • “Mathematically precise, conceptually simple, computationally tractable and intuitively appealing.” • Relevancy is not captured

  20. Query vs. Document Model (a) Query Likelihood (b) Document Likelihood (c) Model Comparison

  21. KL divergence

  22. Thank you.

  23. Questions?

More Related