10601 Machine Learning

1. September 2, 2009 10601 Machine Learning Recitation 2 Öznur Taştan

2. Logistics • Homework 2 is going to be out tomorrow. It is due on Sep 16, Wed. • There is no class on Monday Sep 7th (Labor day) • Those who have not return Homework 1 yet • For details of how to submit the homework policy please check : http://www.cs.cmu.edu/~ggordon/10601/hws.html

3. Outline • We will review • Some probability and statistics • Some graphical models • We will not go over Homework 1 • Since the grace period has not ended yet. • Solutions will be up next week on the web page.

4. We’ll play a game: Catch the goof! • I’ll be the sloppy TA… will make ‘intentional’ mistakes • You’ll catch those mistakes and correct me! Slides with mistakes are marked with Correct slides are marked with

5. Catch the goof!!

6. Law of total probability Given two discrete random variables X and Y X takes values in Y takes values in

7. Law of total probability Given two discrete random variables X and Y X takes values in Y takes values in

8. Law of total probability Given two discrete random variables X and Y X takes values in Y takes values in

9. Law of total probability Given two discrete random variables X and Y X takes values in Y takes values in

10. Law of total probability Given two discrete random variables X and Y Joint probability Marginal probability Conditional probability of X conditioned on Y

11. Law of total probability Given two discrete random variables X and Y Formulas are fine. Anything wrong with the names? Joint probability Marginal probability Conditional probability of X conditioned on Y

12. Law of total probability Given two discrete random variables X and Y Joint probability of X,Y Marginal probability Conditional probability of X conditioned on Y Marginal probability

13. In a strange world Two discrete random variables X and Y take binary values Joint probabilities

14. In a strange world Two discrete random variables X and Y take binary values Joint probabilities Should sum up to 1

15. The world seems fine Two discrete random variables X and Y take binary values Joint probabilities

16. What about the marginals? Joint probabilities Marginal probabilities

17. This is a strange world Joint probabilities Marginal probabilities

18. In a strange world Joint probabilities Marginal probabilities

19. This is a strange world Joint probabilities Marginal probabilities

20. Let’s have a simple problem Joint probabilities Marginal probabilities

21. Conditional probabilities What is the complementary event of P(X=0|Y=1) ? P(X=1|Y=1) OR P(X=0|Y=0)

22. Conditional probabilities What is the complementary event of P(X=0|Y=1) ? P(X=1|Y=1) OR P(X=0|Y=0)

23. The game ends here.

24. Independent number of parameters Assume X and Y take Boolean values {0,1}: • How many independent parameters do you need to fully specify: • marginal probability of X? • the joint probability of P(X,Y)? • the conditional probability of P(X|Y)?

25. Independent number of parameters Assume X and Y take Boolean values {0,1}: • How many independent parameters do you need to fully specify: • marginal probability of X? P(X=0) 1 parameter only [ because P(X=1)+P(X=0)=1 ] • the joint probability of P(X,Y)? P(X=0, Y=0) 3 parameters P(X=0, Y=1) P(X=1, Y=0) • the conditional probability of P(X|Y)?

26. Number of parameters • Assume X and Y take Boolean values {0,1}? • How many independent parameters do you need to fully specify marginal probability of X? P(X=0) 1 parameter only P(X=1)= 1-P(X=0) • How many independent parameters do you need to fully specify the joint probability of P(X,Y)? P(X=0, Y=0) 3 parameters P(X=0, Y=1) P(X=1, Y=0) • How many independent parameters do you need to fully specify the conditional probability of P(X|Y)? P(X=0|Y=0) 2 parameters P(X=0|Y=1)

27. Number of parameters • What about P(X | Y,Z) , how many independent parameters do you need to be able to fully specify the probabilities? Assume each RV takes: m values P(X | Y,Z) q values n values

28. Number of parameters • What about P(X | Y,Z) , how many independent parameters do you need to be able to fully specify the probabilities? Assume each RV takes: m values P(X | Y,Z) q values n values Number of independent parameters: (m-1)*nq

29. Graphical models • A graphical model is a way of representing probabilistic relationships between random variables • Variablesare represented by nodes: • Edges indicates probabilistic relationships: You miss the bus Arrive class late

30. Serial connection Is X and Z independent? ?

31. Serial connection Is X and Z independent? X and Z are not independent

32. Serial connection Is X conditionally independent of Z given Y? ?

33. Serial connection Is X conditionally independent of Z given Y? Yes they are independent

34. How can we show it? Is X conditionally independent of Z given Y?

35. An example case Studied late last night Wake up late Arrive class late

36. Common cause Age Shoe Size Gray Hair X and Y are not marginally independent X and Y are conditionally independent given Z

37. Explaining away Flu Allergy Z X Y Sneeze X and Z marginally independent X and Z conditionally dependent given Y

38. D-separation • X and Z are conditionally independent given Y if Y d-separates X and Z Neither Y nor its descendants should be observed Path between X and Z is blocked by Y

39. D-separation example Is B, C independent given A?

40. D-separation example Is B, C independent given A? Yes

41. D-separation example Observed, A blocks the path Is B, C independent given A? Yes

42. Observed, A blocks the path Is B, C independent given A? Yes not observed neither its descendants

43. D-separation example Is A, F independent given E?

44. Is A, F independent given E? Yes

45. Is A, F independent given E? Yes

46. Is C, D independent given F?

47. Is C, D independent given F? No

48. Is A, G independent given B and F?

49. Is A, G independent given B and F? Yes

50. Naïve Bayes Model J D C R J: The person is a junior D: The person knows calculus C: The person leaves in campus R: Saw the “Return of the King” more than once