Mean Field Inference in Dependency Networks: An Empirical Study

1. Mean Field Inference in Dependency Networks: An Empirical Study Daniel Lowd and ArashShamaei University of Oregon

2. Learning and Inference inGraphical Models We want to learn a probability distribution from data and use it to answer queries. Applications: medical diagnosis, fault diagnosis, web usage analysis, bioinformatics, collaborative filtering, etc. Data Model Answers! A B C Learning Inference

3. One-Slide Summary • In dependency networks, mean field inference is faster than Gibbs sampling, with similar accuracy. • Dependency networks are competitive with Bayesian networks. Data Model Answers! A B C Learning Inference

4. Outline • Graphical models:Dependency networks vs. others • Representation • Learning • Inference • Mean field inference in dependency networks • Experiments

5. Dependency Networks [Heckerman et al., 2000] Represents a probability distribution over {X1, …, Xn} as a set of conditional probability distributions. Example: X1 X2 X3

6. Comparison of Graphical Models

7. B=? true false <0.2, 0.8> C=? true false <0.5, 0.5> <0.7, 0.3> Learning Dependency Networks [Heckerman et al., 2000] For each variable Xi, learn conditional distribution, C=? true false <0.7, 0.3> <0.4, 0.6>

8. Approximate Inference Methods • Gibbs sampling: Slow but effective • Mean field: Fast and usually accurate • Belief propagation: Fast and usually accurate Model Answers! A B C

9. Gibbs Sampling Resample each variable in turn, given its neighbors: Use set of samples to answer queries.e.g., Converges to true distribution, given enough samples (assuming positive distribution). Previously, the only method used to compute probabilities in DNs.

10. Mean Field Approximate P with simpler distribution Q: To find best Q, optimize reverse K-L divergence: Mean field updates converge to local optimum: Works for DNs! Never before tested!

11. Mean Field in Dependency Networks • Initialize each Q(Xi) to a uniform distribution. • Update each Q(Xi) in turn: • Stop when marginalsQ(Xi) converge. If consistent, this is guaranteed to converge. If inconsistent, this always seems to converge in practice.

12. Empirical Questions Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy? Q2. How do DNs compare to BNs in inference speed and accuracy?

13. Experiments • Learned DNs and BNs on 12 datasets • Generated queries from test data • Varied evidence variables from 10% to 90% • Score using average CMLL per variable(conditional marginal log-likelihood):

14. Results: Accuracy in DNs Negative CMLL

15. Results: Timing in DNs (log scale)

16. MF vs. Gibbs in DNs,run for equal time In DNs, MF usually more accurate, given equal time.

17. Results: Accuracy

18. Gibbs: DN vs. BN With more evidence, DNs are more accurate.

19. Experimental Results Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy? A1. MF is consistently faster with similar accuracy, or more accurate with similar speed. Q2. How do DNs compare to BNs in inference speed and accuracy? A2.DNs are competitive with BNs – better with more evidence, worse with less evidence.

20. Conclusion • MF inference in DNs is fast and accurate, especially with more evidence. • Future work: • Relational dependency networks (Neville & Jensen, 2007) • More powerful approximations Source code available: http://libra.cs.uoregon.edu/

21. Results: Timing (log scale)

22. Learned Models Learning time is comparable. DNs usually have higher pseudo-likelihood (PLL) DNs sometimes have higher log-likelihood (LL)