Information Retrieval

Information Retrieval Chap. 02: Modeling - Part 2 Slides from the text book author, modified by L N Cassel September 2003

Probabilistic Model • Objective: to capture the IR problem using a probabilistic framework • Given a user query, there is an ideal answer set • Querying as specification of the properties of this ideal answer set (clustering) • But, what are these properties? • Guess at the beginning what they could be (i.e., guess initial description of ideal answer set) • Improve by iteration

Probabilistic Model • An initial set of documents is retrieved somehow • User inspects these docs looking for the relevant ones (in truth, only top 10-20 need to be inspected) • IR system uses this information to refine description of ideal answer set • By repeating this process, it is expected that the description of the ideal answer set will improve • Have always in mind the need to guess at the very beginning the description of the ideal answer set • Description of ideal answer set is modeled in probabilistic terms

Probabilistic Ranking Principle • Given a user query q and a document dj, the probabilistic model tries to estimate the probability that the user will find the document dj interesting (i.e., relevant). • The model assumes that this probability of relevance depends on the query and the document representations only. • Ideal answer set is referred to as R and should maximize the probability of relevance. Documents in the set R are predicted to be relevant. • But, • how to compute probabilities? • what is the sample space?

The Ranking • Probabilistic ranking computed as: sim(q,dj) = P(dj relevant-to q) / P(dj non-relevant-to q) This is the odds of the document dj being relevant Taking the odds minimizes the probability of an erroneous judgement • Definition: wij  {0,1} P(R | dj) :probability that given document is relevant P(R | dj) : probability document is not relevant

The Ranking • sim(dj,q) = P(R | dj) / P(R | dj) = [P( dj | R) * P(R)] [P( dj | R) * P(R)] ~ P( dj | R) P( dj | R) • P( dj | R) : probability of randomly selecting the document dj from the set R of relevant documents • P(R): probability that a document selected at random from the whole set of documents is relevant. P(R) and P(R) are the same.

Improving the Initial Ranking • sim(dj,q) ~ ~  wiq * wij * (log P(ki | R) + log P(ki | R) ) P(ki | R) P(ki | R) • Let • V : set of docs initially retrieved • Vi : subset of docs retrieved that contain ki • Reevaluate estimates: • P(ki | R) = Vi V • P(ki | R) = ni - Vi N - V • Repeat recursively

Pluses and Minuses • Advantages: • Docs ranked in decreasing order of probability of relevance • Disadvantages: • need to guess initial estimates for P(ki | R) • method does not take into account tf and idf factors

Brief Comparison of Classic Models • Boolean model does not provide for partial matches and is considered to be the weakest classic model • Salton and Buckley did a series of experiments that indicate that, in general, the vector model outperforms the probabilistic model with general collections • This seems also to be the view of the research community

Set Theoretic Models • The Boolean model imposes a binary criterion for deciding relevance • The question of how to extend the Boolean model to accomodate partial matching and a ranking has attracted considerable attention in the past • We discuss now two set theoretic models for this: • Fuzzy Set Model • Extended Boolean Model (We will not discuss because of time limitations)

Fuzzy Set Model • Queries and docs represented by sets of index terms: matching is approximate from the start • This vagueness can be modeled using a fuzzy framework, as follows: • with each term is associated a fuzzy set • each doc has a degree of membership in this fuzzy set • This interpretation provides the foundation for many models for IR based on fuzzy theory • In here, we discuss the model proposed by Ogawa, Morita, and Kobayashi (1991)

Fuzzy Set Theory • Framework for representing classes whose boundaries are not well defined • Key idea is to introduce the notion of a degree of membership associated with the elements of a set • This degree of membership varies from 0 to 1 and allows modeling the notion of marginal membership • Thus, membership is now a gradual notion, contrary to the crispy notion enforced by classic Boolean logic

Fuzzy Set Theory • Definition • A fuzzy subset A of U is characterized by a membership function (A,u) : U  [0,1] which associates with each element u of U a number (u) in the interval [0,1] • Definition • Let A and B be two fuzzy subsets of U. Also, let ¬A be the complement of A. Then, • (¬A,u) = 1 - (A,u) • (AB,u) = max((A,u), (B,u)) • (AB,u) = min((A,u), (B,u))

Fuzzy Information Retrieval • Fuzzy sets are modeled based on a thesaurus • This thesaurus is built as follows: • Let be a term-term correlation matrix • Let c(i,l) be a normalized correlation factor for (ki,kl): c(i,l) = n(i,l) ni + nl - n(i,l) • ni: number of docs which contain ki • nl: number of docs which contain kl • n(i,l): number of docs which contain both ki and kl • We now have the notion of proximity among index terms. c

Fuzzy Information Retrieval • The correlation factor c(i,l) can be used to define fuzzy set membership for a document dj as follows: (i,j) = 1 -  (1 - c(i,l)) ki  dj (i,j) : membership of doc dj in fuzzy subset associated with ki • The above expression computes an algebraic sum over all terms in the doc dj (shown here as complement of negated algebraic product) • A doc dj belongs to the fuzzy set for ki, if its own terms are associated with ki

Fuzzy Information Retrieval • (i,j) = 1 -  (1 - c(i,l)) ki  dj • (i,j) : membership of doc dj in fuzzy subset associated with ki • If doc dj contains a term kl which is closely related to ki, we have • c(i,l) ~ 1 (correlation between terms i, I is high) • (i,j) ~ 1 (membership of term i in document j is high) • index ki is a good fuzzy index for document j.

Ka Kb cc2 cc3 cc1 Kc Fuzzy IR: An Example • q = ka  (kb  kc) • = (1,1,1) + (1,1,0) + (1,0,0) = cc1 + cc2 + cc3 • (q,dj) = (cc1+cc2+cc3,j) = 1 - (1 - (a,j) (b,j) (c,j)) * (1 - (a,j) (b,j) (1-(c,j))) * (1 - (a,j) (1-(b,j)) (1-(c,j))) qdnf (1,1,1) (1,1,0) (1,0,0) Exercise - put some numbers in the areas and calculate the actual value of (q,dj)

Fuzzy Information Retrieval • Fuzzy IR models have been discussed mainly in the literature associated with fuzzy theory • Experiments with standard test collections are not available • Difficult to compare at this time

Alternative Probabilistic Models • Probability Theory • Semantically clear • Computationally clumsy • Why Bayesian Networks? • Clear formalism to combine evidences • Modularize the world (dependencies) • Bayesian Network Models for IR • Inference Network (Turtle & Croft, 1991) • Belief Network (Ribeiro-Neto & Muntz, 1996)

Bayesian Inference Basic Axioms: • 0 < P(A) < 1 ; • P(sure)=1; • P(A V B)=P(A)+P(B) if A and B are mutually exclusive

Bayesian Inference Other formulations • P(A)=P(A  B)+P(A  ¬B) • P(A)= i P(A  Bi) , where Bi,i is a set of exhaustive and mutually exclusive events • P(A) + P(¬A) = 1 • P(A|K) belief in A given the knowledge K • if P(A|B)=P(A), we say:A and B are independent • if P(A|B C)= P(A|C), we say: A and B are conditionally independent, given C • P(A  B)=P(A|B)P(B) • P(A)= i P(A | Bi)P(Bi)

Bayesian Inference Bayes’ Rule : the heart of Bayesian techniques P(H|e) = P(e|H)P(H) / P(e) Where, H : a hypothesis and e is an evidence P(H) : prior probability P(H|e) : posterior probability P(e|H) : probability of e if H is true P(e) : a normalizing constant, then we write: P(H|e) ~ P(e|H)P(H)

Bayesian Networks Definition: Bayesian networks are directed acyclic graphs (DAGS) in which the nodes represent random variables, the arcs portray causal relationships between these variables, and the strengths of these causal influences are expressed by conditional probabilities.

Bayes - resource • Look at • http://members.aol.com/johnp71/bayes.html

Bayesian Networks yt y1 y2 … yi : parent nodes (in this case, root nodes) x : child node yi cause x Y the set of parents of x The influence of Y on x can be quantified by any function F(x,Y) such that x F(x,Y) = 1 0 < F(x,Y) <1 For example, F(x,Y)=P(x|Y) x

Bayesian Networks x1 Given the dependencies declared in a Bayesian Network, the expression for the joint probability can be computed as a product of local conditional probabilities, for example, P(x1, x2, x3, x4, x5)= P(x1 ) P(x2| x1 ) P(x3| x1 ) P(x4| x2, x3 ) P(x5| x3 ). P(x1 ) : prior probability of the root node x2 x3 x5 x4

Bayesian Networks x1 In a Bayesian network each variable x is conditionally independent of all its non-descendants, given its parents. For example: P(x4, x5| x2 ,x3)= P(x4| x2 ,x3) P( x5| x3) x3 x2 x4 x5

Inference Network Model • Epistemological view of the IR problem • Random variables associated with documents, index terms and queries • A random variable associated with a document dj represents the event of observing that document

Inference Network Model Nodes documents (dj) index terms (ki) queries (q, q1, and q2) user information need (I) Edges from dj to its index term nodes ki indicate that the observation of dj increase the belief in the variables ki .

Inference Network Model dj has index terms k2, ki, and kt q has index terms k1, k2, and ki q1 and q2 model boolean formulation q1=((k1 k2) v ki); I = (q v q1)

Inference Network Model Definitions: k1, dj,, and q random variables. k=(k1, k2, ...,kt) a t-dimensional vector ki,i{0, 1}, then k has 2t possible states dj,j{0, 1}; q{0, 1} The rank of a document dj is computed as P(q dj) q and dj,are short representations for q=1 and dj =1 (dj stands for a state where dj = 1 and ljdl =0, because we observe one document at a time)

Inference Network Model P(q  dj) = k P(q  dj| k) P(k) = k P(q  dj  k) = k P(q | dj  k) P(dj  k) = k P(q | k) P(k | dj ) P( dj ) P(¬(q  dj)) = 1 - P(q  dj)

Inference Network Model The prior probability P(dj) reflects the probability associated to the event of observing a given document dj • Uniformly for N documents P(dj) = 1/N P(¬dj) = 1 - 1/N • Based on norm of the vector dj P(dj)= 1/|dj| P(¬dj) = 1 - 1/|dj|

Inference Network Model For the Boolean Model P(dj) = 1/N P(ki | dj) = 1 if gi(dj)=1 0 otherwise P(¬ki | dj) = 1 - P(ki | dj)  only nodes associated with the index terms of the document dj are activated

Inference Network Model For the Boolean Model 1 if qcc | (qcc qdnf)  ( ki, gi(k)= gi(qcc) P(q | k) = 0 otherwise P(¬q | k) = 1 - P(q | k)  one of the conjunctive components of the query must be matched by the active index terms in k

Inference Network Model For a tf-idf ranking strategy P(dj)= 1 / |dj| P(¬dj) = 1 - 1 / |dj|  prior probability reflects the importance of document normalization

Inference Network Model For a tf-idf ranking strategy P(ki | dj) = fi,j P(¬ki | dj)= 1- fi,j  the relevance of the a index term ki is determined by its normalized term-frequency factorfi,j =freqi,j / max freql,j

Inference Network Model For a tf-idf ranking strategy Define a vector ki given by ki = k | ((gi(k)=1)  (ji gj(k)=0))  in the state ki only the node ki is active and all the others are inactive

Inference Network Model For a tf-idf ranking strategy idfi if k = ki  gi(q)=1 P(q | k) = 0 if k  ki v gi(q)=0 P(¬q | k) = 1 - P(q | k)  we can sum up the individual contributions of each index term by its normalized idf

Inference Network Model For a tf-idf ranking strategy Applying the previous probabilities we have P(q  dj) = Cj (1/|dj|) i [fi,j idfi (1/(1- fi,j ))]  Cj vary from document to document  the ranking is distinct of the one provided by the vector model

Inference Network Model Combining evidential source Let I = q v q1 P(I  dj) = k P(I | k) P(k | dj ) P( dj) = k [1 - P(¬q|k)P(¬q1| k)] P(k| dj ) P( dj)  it might yield a retrieval performance which surpasses the retrieval performance of the query nodes in isolation (Turtle & Croft)

Belief Network Model • As the Inference Network Model • Epistemological view of the IR problem • Random variables associated with documents, index terms and queries • Contrary to the Inference Network Model • Clearly defined sample space • Set-theoretic view • Different network topology

Belief Network Model The Probability Space Define: K={k1, k2, ...,kt} the sample space (a concept space) u  K a subset of K (a concept) ki an index term (an elementary concept) k=(k1, k2, ...,kt) a vector associated to each u such that gi(k)=1  ki  u ki a binary random variable associated with the index term ki , (ki = 1 gi(k)=1  ki  u)

Belief Network Model A Set-Theoretic View Define: a document dj and query q as concepts in K a generic concept c in K a probability distribution P over K, as P(c)=uP(c|u) P(u) P(u)=(1/2)t P(c) is the degree of coverage of the space K by c

Information Retrieval