Bayesian networks
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Bayesian Networks PowerPoint PPT Presentation


  • 115 Views
  • Uploaded on
  • Presentation posted in: General

Bayesian Networks. Motivation. The conditional independence assumption made by naïve Bayes classifiers may seem to rigid, especially for classification problems in which the attributes are somewhat correlated.

Download Presentation

Bayesian Networks

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Bayesian networks

Bayesian Networks


Motivation

Motivation

  • The conditional independence assumption made by naïve Bayes classifiers may seem to rigid, especially for classification problems in which the attributes are somewhat correlated.

  • We talk today for a more flexible approach for modeling the conditional probabilities.


Na ve bayes and correlated attrs

Naïve Bayes and Correlated Attrs

  • Two binary attributes A and B, and a binary class Y.

  • A distribution

    • P(A=0 | Y=0) = 0.4P(A=1 | Y=0) = 0.6

    • P(A=0 | Y=1) = 0.6P(A=1 | Y=1) = 0.4

  • B is perfectly correlated with A when Y=0, but not when Y=1

    • P(B=0 | Y=0) = 0.4P(B=1 | Y=0) = 0.6

    • P(B=0 | Y=1) = 0.5P(B=1 | Y=1) = 0.5

  • Also, let’s assume

    • P(Y=0) = P(Y=1) = 0.5


Na ve bayes and correlated attrs1

Naïve Bayes and Correlated Attrs

Now, we are given a new record with A=0 and B=0.

Using Naïve Bayes we have:

P(Y=0 | A=0, B=0) =  * P(A=0 | Y=0) * P(B=0 | Y=0) * P(Y=0) =  * .4 * .4 * .5 =  * 0.08

P(Y=1 | A=0, B=0) =  * P(A=0 | Y=1) * P(B=0 | Y=1) * P(Y=1) =  * .6 * .5 * .5 =  * .15

So, we predict Y=1.


Na ve bayes and correlated attrs2

Naïve Bayes and Correlated Attrs

  • However, since A and B are perfectly correlated (when Y=0) we have that:

    P(A=0, B=0 | Y=0) = P(A=0 | Y=0) = 0.4

  • Thus,

    P(Y=0 | A=0, B=0) =  * P(A=0, B=0 | Y=0) * P(Y=0)

    =  * 0.4 * 0.5

    =  * 0.2

    which is greater than

    P(Y=1 | A=0, B=0) =  * P(A=0 | Y=1) * P(B=0 | Y=1) * P(Y=1) =  * .6 * .5 * .5 =  * .15

    So, the record should have been classified as class 0.


Bayesian networks1

Bayesian networks

  • A simple, graphical notation for conditional independence assertions.

    Syntax:

  • a set of nodes, one per variable (attribute)

  • a directed, acyclic graph (link means: "directly influences")

  • a conditional distribution for each node given its parents:

    P (Xi | Parents (Xi))

  • The conditional distribution is represented as a conditional probability table (CPT) giving the distribution over Xi for each combination of parent values.


Example perls example

Example (Perls’ example)

  • I'm at work, neighbor John calls to say my alarm is ringing, but neighbor Mary doesn't call. Sometimes it's set off by minor earthquakes. Is there a burglar?

  • John always calls when he hears the alarm, but sometimes confuses the telephone ringing with the alarm.

  • Mary likes rather loud music and sometimes misses the alarm.

  • Variables: Burglary, Earthquake, Alarm, JohnCalls, MaryCalls

  • Network topology reflects "causal" knowledge:

    • A burglar can set the alarm off

    • An earthquake can set the alarm off

    • The alarm can cause Mary to call

    • The alarm can cause John to call


Example cont d

Example cont’d

To save space, some of

the probabilities have been omitted from the diagram. The omitted probabilities can be recovered by noting that P(X = x)= 1 - P(X =x)and

P(X = x|Y)= 1 - P(X=x|Y), where x denotes the opposite outcome of x.

The topology shows that burglary and earthquakes directly affect the probability of alarm, but whether Mary or John call depends only on the alarm.

Thus our assumptions are that they don’t perceive any burglaries directly,

and they don’t confer before calling.


Semantics

Semantics

Suppose we have the variables X1,…,Xn.

The probability for them to have the values x1,…,xn respectively is P(xn,…,x1):

e.g.,

P(j  m  a b e)

= P(j | a) P(m | a) P(a | b, e) P(b) P(e)

= …

P(xn,…,x1):

is short for

P(Xn=xn,…, Xn= x1):


Inference in bayesian networks

Inference in Bayesian Networks

  • The basic task for a probabilistic inference system is to compute the posterior probability for a query variable (class attribute), given some observed event

    • that is, some assignment of values to a set of evidence variables (some of the other attributes).

  • Notation:

    • X denotes query variable

    • E denotes the set of evidence variables E1,…,Em, and e is a particular event, i.e. an assignment to the variables in E.

    • Y will denote the set of the remaining variables (hidden variables).

  • A typical query asks for the posterior probability P(x|e1,…,em)

  • E.g. We could ask: What’s the probability of a burglary if both Mary and John call, P(burglary | johhcalls, marycalls)?


Classification

Classification

  • Suppose, we are given for the evidence variables E1,…,Em, their values e1,…,em, and we want to predict whether the query variable X has the value x or not.

  • For this we compute and compare the following:

  • However, how do we compute:

What about the hidden variables Y1,…,Yk?


Inference by enumeration

Inference by enumeration

Example:P(burglary | johhcalls, marycalls)? (Abbrev. P(b|j,m))

In general:


Numerically

Numerically…

P(b | j,m) = P(b) aP(j|a)P(m|a)eP(a|b,e)P(e) = …=  * 0.00059

P(b | j,m) = P(b) aP(j|a)P(m|a)eP(a| b,e)P(e) = …=  * 0.0015

P(B | j,m) =  <0.00059, 0.0015> = <0.28, 0.72>.


P b j m

P(b | j,m)

P(b | j,m) = P(b) aP(j|a)P(m|a)eP(a|b,e)P(e)

= P(b) aP(j|a)P(m|a)(P(a|b,e)P(e) + P(a|b,e)P(e))

= P(b)( P(j|a)P(m|a)( P(a|b,e)P(e) + P(a|b,e)P(e) )

+ P(j|a)P(m|a)( P(a|b,e)P(e) + P(a|b,e)P(e) ))

=  * .001*(.9*.7*(.95*.002 + .94*.998) +.05*.01*(.05*.002 + .71*.998) )

=  * .00059


Bayesian networks

P(b | j,m)

  • P(b | j,m) = P(b) aP(j|a)P(m|a)eP(a|b,e)P(e)

  • = P(b) aP(j|a)P(m|a)(P(a|b,e)P(e) + P(a|b,e)P(e))

  • = P(b)( P(j|a)P(m|a)( P(a|b,e)P(e) + P(a|b,e)P(e) )

  • + P(j|a)P(m|a)( P(a|b,e)P(e) + P(a|b,e)P(e) ))

  • =  * .999*(.9*.7*(.29*.002 + .001*.998) +.05*.01*(.71*.002 + .999*.998) )

  • =  * .0015

  • = 1/(.00059 + .0015)

    = 478.5

    P(b | j,m) = 478.5 * .00059

    =.28

    P(b | j,m) = 478.5 * .0015

    =.72


Constructing bayesian networks

Constructing Bayesian networks

1. Choose an ordering of variables X1, … ,Xn

2. For i = 1 to n

  • add Xi to the network

  • select parents from X1, … ,Xi-1 such that

    P(Xi | Parents(Xi)) = P(Xi | X1, ... Xi-1)

    This choice of parents guarantees:

    P(X1, … ,Xn) = i =1P(Xi | X1, … , Xi-1) (chain rule)

    = i =1P(Xi | Parents(Xi)) (by construction)

  • Choosing the parents from X1, … , Xi-1 is done by domain human experts.


  • Example

    Example

    • The ordering of variables is very important.

    • E.g. suppose we choose the ordering M, J, A, B, E

      Adding MaryCalls: No parents

      P(J|M) = P(J)?

      Is P(John calling) independent of P(Mary calling)?

      Clearly not, since, on any given day, if Mary called, then the probability that John called is much better than the background probability that he called.

      So, we add a link from MaryCalls to JohnCalls.


    Example1

    Example

    • Suppose we choose the ordering M, J, A, B, E

      Adding the A (Alarm) node: Is

      P(A | J, M) = P(A | J)?

      P(A | J, M) = P(A)?

      No.

      Clearly, if both call, it’s more likely that the alarm has gone off that if just one or neither call, so we need both MaryCalls and JohnCalls as parents.


    Example2

    Example

    • Suppose we choose the ordering M, J, A, B, E

      Adding B (Burglary) node: Is

      P(B | A, J, M) = P(B | A)?

      P(B | A, J, M) = P(B)?

      Yes for the first. No for the second.

      If we know the alarm state, then the call from John or Mary might give us information about the phone ringing or Mary’s music, but not about burglary.

      So, we need just Alarm as parent.


    Example3

    Example

    • Suppose we choose the ordering M, J, A, B, E

    • Adding E (Earthquake) node: Is

      P(E | B, A ,J, M) = P(E | A)?

      P(E | B, A, J, M) = P(E | A, B)?

      No for the first. Yes for the second.

      If the alarm is on, it is more likely that there has been an earthquake.

      But if we know there has been a burglary, then that explains the alarm, and the probability of an earthquake would be only slightly above normal.

      Hence we need both Alarm and Burglary as parents.


    Example cont d1

    Example cont’d

    • So, the network is less compact if we go non-causal: 1 + 2 + 4 + 2 + 4 = 13 numbers needed instead of 10 if we go in causal direction.

    • Deciding conditional independence is harder in noncausal directions

    • Causal models and conditional independence seem hardwired for humans!


  • Login