Bayesian Networks Chapter 2 (Duda et al.) – Section 2.11

1. Bayesian NetworksChapter 2 (Duda et al.) – Section 2.11 CS479/679 Pattern RecognitionDr. George Bebis

2. Bayesian Networks - Motivation • High-dimensional densities are very challenging to model since they depend on many parameters (e.g., knvalues) • Having knowledge of which variables are (orare not)dependent could be very useful. • Such dependencies can be represented efficiently using Bayesian Networks (or Belief Networks).

3. Example of Dependencies • Model the state of an automobile: • Engine temperature • Brake fluid pressure • Coolant temperature • Tire air pressure • Wire voltages etc. • Causally related variables • Engine temperature • Coolant temperature • NOT causally related variables • Engine temperature • Tire air pressure

4. Bayesian Nets - Applications • Microsoft: Answer Wizard, Print Troubleshooter http://erichorvitz.com/ftp/x2xins.lo.pdf • US Army: SAIP (Battalion Detection from SAR, IR etc.) • NASA: Vista (DSS for Space Shuttle) • GE: Gems (real-time monitoring of utility generators)

5. Definitions • A Bayesian net is typically represented by a Directed Acyclic Graph (DAG) • Each node represents a variable. • Each variable assumes certain values (i.e., states).

6. Dependencies Between Nodes • Links between nodes are directional and represent a causal influence (e.g., A influences X or X depends on A) • Influences could be direct or indirect (e.g., A influences X directly and A influences C indirectly through X).

7. Prior / Conditional Probabilities • Each variable is associated with prior or conditional probabilities (discrete or continuous).

8. Markov Property “Each node is conditionally independent of its ancestors given its parents” Example: parents of x1

9. Computing Joint ProbabilitiesUsing the Markov property • Using the chain rule, the joint probability of a set of variables x1, x2, …, xn is given as: • Using the Markov property (i.e., node xi is conditionally independent of its ancestors given its parents πi), we have : = much simpler!

10. Example • We can compute the probability of any configuration of states in the joint density, e.g.: P(a3, b1, x2, c3, d2)=P(a3)P(b1)P(x2 /a3,b1)P(c3 /x2)P(d2 /x2)= 0.25 x 0.6 x 0.4 x 0.5 x 0.4 = 0.012

11. Fundamental Problems in Bayesian Nets • Evaluation (inference): Given the values of the observed variables (evidence), estimate the values of the non-observed variables. • Learning: Given training data and prior information (e.g., expert knowledge, causal relationships), estimate the network structure, or the parameters (probabilities), or both.

12. Example: Medical Diagnosis Uppermost nodes:biological agents (bacteria, virus) Intermediate nodes:diseases Lowermost nodes:symptoms • Goal: given some evidence (biological agents, symptoms), find most likely disease. causes effects

13. Example: Medical Diagnosis

14. Evaluation (Inference) Problem • In general, if X denotes the query variables and e denotes the evidence, then where α=1/P(e) is a constant of proportionality. joint density

15. Example • Classify a fish given that it is light (c1) and was caught in south Atlantic (b2) -- no evidence about what time of the year the fish was caught nor its thickness.

16. Example (cont’d) marginalize Markov property

17. Example (cont’d)

18. Example (cont’d) • Similarly, P(x2 / c1,b2)=α 0.066 • Normalize probabilities (not necessary): P(x1 /c1,b2)+ P(x2 /c1,b2)=1 (α=1/0.18) P(x1 /c1,b2)= 0.73 P(x2 /c1,b2)= 0.27 salmon

19. Evaluation (Inference) Problem (cont’d) • Exact inference is an NP-hard problembecause the number of terms in the summations (or integrals) for discrete (or continuous) variables grows exponentially with the number of variables. • For some restricted classes of networks (e.g., singly connected networks where there is no more than one path between any two nodes) exact inference can be efficiently solved in time linear in the number of nodes.

20. Evaluation (Inference) Problem (cont’d) • For singly connected Bayesian networks: • Approximate inference methods are typically used in most cases. • Sampling (Monte Carlo) methods • Variational methods • Loopy belief propagation

21. Another Example • You have a new burglar alarm installed at home. • It is fairly reliable at detecting burglary, but also sometimes responds to earthquakes. • You have two neighbors, Ali and Veli, who promised to call you at work when they hear the alarm.

22. Another Example (cont’d) • Ali always calls when he hears the alarm, but sometimes confuses telephone ringing with the alarm and calls too. • Veli likes loud music and sometimes misses the alarm. • Design a Bayesian network to estimate: P( burglary/evidence)

23. Another Example (cont’d) • What are the problem variables? • Alarm • Causes • Burglary, Earthquake • Effects • Ali calls, Veli calls

24. Another Example (cont’d) • What are the conditional dependencies among them? • Burglary (B) and earthquake (E) directly affect the probability of the alarm (A) going off • Whether or not Ali calls (AC) or Veli calls (VC) depends on the alarm.

25. Another Example (cont’d)

26. Another Example (cont’d) • What is the probability that the alarm has sounded but neither a burglary nor an earthquake has occurred, and both Ali and Veli call?

27. Another Example (cont’d) • What is the probability that there is a burglary given that Ali calls? • What about if both Veli and Ali call?

28. Naïve Bayesian Network • Assuming that features are conditionally independent, the conditional class density can be simplified as follows: • Sometimes works well in practice despite the strong assumption of independence. Naïve Bayesian Network: