第十讲 概率图模型导论 Chapter 10 Introduction to Probabilistic Graphical Models

1. 浙江大学计算机学院《人工智能引论》课件 第十讲 概率图模型导论Chapter 10 Introduction to Probabilistic Graphical Models Weike Pan, andCongfu Xu {panweike, xucongfu}@zju.edu.cn Institute of Artificial Intelligence College of Computer Science, Zhejiang University October 12, 2006

2. References • An Introduction to Probabilistic Graphical Models. Michael I. Jordan. • http://www.cs.berkeley.edu/~jordan/graphical.html

3. Outline • Preparations • Probabilistic Graphical Models (PGM) • Directed PGM • Undirected PGM • Insights of PGM

4. Outline • Preparations • PGM “is” a universal model • Different thoughts of machine learning • Different training approaches • Different data types • Bayesian Framework • Chain rules of probability theory • Conditional Independence • Probabilistic Graphical Models (PGM) • Directed PGM • Undirected PGM • Insights of PGM

5. Different thoughts of machine learning • Statistics (modeling uncertainty, detailed information) vs. Logics (modeling complexity, high level information) • Unifying Logical and Statistical AI. Pedro Domingos, University of Washington. AAAI 2006. • Speech: Statistical information (Acoustic model + Language model + Affect model…) + High level information (Expert/Logics)

6. Different training approaches • Maximum Likelihood Training: MAP (Maximum a Posteriori) vs. Discriminative Training: Maximum Margin (SVM) • Speech: classical combination – Maximum Likelihood + Discriminative Training

7. Different data types • Directed acyclic graph (Bayesian Networks, BN) • Modeling asymmetric effects and dependencies: causal/temporal dependence (e.g. speech analysis, DNA sequence analysis…) • Undirected graph (Markov Random Fields, MRF) • Modeling symmetric effects and dependencies: spatial dependence (e.g. image analysis…)

8. PGM “is” a universal model • To model both temporal and spatial data, by unifying • Thoughts: Statistics + Logics • Approaches: Maximum Likelihood Training + Discriminative Training • Further more, the directed and undirected models together provide modeling power beyond that which could be provided by either alone.

9. Bayesian Framework Problem description Observation  Conclusion (classification or prediction) Bayesian rule Likelihood Priori probability Observation A posteriori probability Class i Normalization factor What we care is the conditional probability, and it’s is a ratio of two marginal probabilities.

10. Chain rules of probability theory

11. Conditional Independence

12. Outline • Preparations • Probabilistic Graphical Models (PGM) • Directed PGM • Undirected PGM • Insights of PGM

13. PGM • Nodes represent random variables/states • The missing arcs represent conditional independenceassumptions • The graph structure impliesthe decomposition

14. Directed PGM (BN) Queries Probability Distribution Representation Implementation Conditional Independence Interpretation

15. Probability Distribution Definition of Joint Probability Distribution Check:

16. Representation Graphical models represent joint probability distributions more economically, using a set of “local” relationships among variables.

17. Conditional Independence (basic) Interpret missing edges in terms of conditional independence • Assert the conditional independence of a node from its ancestors, conditional on its parents.

18. Conditional Independence (3 canonical graphs) Conditional Independence Marginal Independence Classical Markov chain “Past”, “present”, “future” Common cause Y “explains” all the dependencies between X and Z Common effect Multiple, competing explanation

19. Conditional Independence (check) Bayes ball algorithm (rules) One incoming arrow and one outgoing arrow Two outgoing arrows Two incoming arrows Check through reachability

20. Outline • Preparations • Probabilistic Graphical Models (PGM) • Directed PGM • Undirected PGM • Insights of PGM

21. Undirected PGM (MRF) Queries Probability Distribution Representation Implementation Conditional Independence Interpretation

22. Probability Distribution(1) • Clique • A clique of a graph is a fully-connected subset of nodes. • Local functions should not be defined on domains of nodes that extend beyond the boundaries of cliques. • Maximal cliques • The maximal cliques of a graph are the cliques that cannot be extended to include additional nodes without losing the probability of being fully connected. • We restrict ourselves to maximal cliques without loss of generality, as it captures all possible dependencies. • Potential function (local parameterization) • : potential function on the possible realizations of the maximal clique

23. Probability Distribution(2) Maximal cliques

24. Probability Distribution(3) • Joint probability distribution Normalization factor Boltzman distribution

25. Conditional Independence It’s a “reachability” problem in graph theory.

26. Representation

27. Outline • Preparations • Probabilistic Graphical Models (PGM) • Directed PGM • Undirected PGM • Insights of PGM

28. Insights of PGM (Michael I. Jordan) • Probabilistic Graphical Models are a marriage between probability theory and graph theory. • A graphical model can be thought of as a probabilistic database, a machine that can answer “queries” regarding the values of sets of random variables. • We build up the database in pieces, using probability theory to ensure that the pieces have a consistent overall interpretation. Probability theory also justifies the inferential machinery that allows the pieces to be put together “on the fly” to answer the queries. • In principle, all “queries” of a probabilistic database can be answered if we have in hand the joint probability distribution.

29. Insights of PGM (data structure & algorithm) • A graphical model is a natural/perfect tool for • representation(数据结构) and • inference (算法).

30. Thanks!