Review of Probability Theory

1. Review of Probability Theory Fall 2014 The University of Iowa Tianbao Yang

2. Announcements • Office hours of TA changed • Mon, Wed. 3:30-5:00pm • Materials available online

3. A Question If you choose an answer to this question at random, what is the chance you will be correct? 25% 50% 60% 25%

4. Today’s Topics • Basic Concepts • Two Rules of Probability • Bayes’ Theorem • Distributions and Statistics • Data Likelihood and MLE Pattern Recognition and Machine Learning Chapter 2

5. Experiments, Sample Space, Events • Experiment: an activity with observable outcomes e.g. rolling a dice once • Sample Space: • all possible outcomes • An event: • a subset of the sample space • getting a value one S E

6. Probability • Probability of an event: a measure of likeliness • if all outcomes are equally possible • Three Axioms • If not, repeat the experiment infinite times (frequentist probability) S E1 E2

7. A Question Which ball is more likely to be selected? 1 2 3 4 1 2 5 3 4 5 1 2 1 5 1 2 3 4 2 3 4 3 4 5 5 Red box (0.5) Blue box (0.5)

8. Today’s Topics • Basic Concepts • Two Rules of Probability • Bayes’ Theorem • Distributions and Statistics • Data Likelihood and MLE Pattern Recognition and Machine Learning Chapter 2

9. Random variable • Random variable: a variable whose value is subject to variations due to chance • Rolling a dice: • Probability of a random variable

10. Two Rules of Probability • Consider two random variables 1 1 1

11. Two Rules of Probability • Consider two random variables • N times of experiments Row Sum Column Sum

12. Two Rules of Probability • Probabilities • Marginal probability of X • Marginal probability of Y

13. Two Rules of Probability • Probabilities • Marginal probability of X • Marginal probability of Y • Joint probability of X and Y

14. Two Rules of Probability • Probabilities • Marginal probability of X • Marginal probability of Y • Joint probability of X and Y • Conditional probability of Y given X

15. Two Rules of Probability • Sum Rule marginal probability joint probability

16. Two Rules of Probability • Product rule joint probability conditional probability marginal probability

17. Two Rules of Probability Which ball is more likely to be selected? Marginal probabilities Red box(0.5) Blue box(0.5) Conditional probabilities

18. Today’s Topics • Basic Concepts • Two Rules of Probability • Bayes’ Theorem • Distributions and Statistics • Data Likelihood and MLE Pattern Recognition and Machine Learning Chapter 2

19. Bayes’ Theorem • Product rule Thomas Bayes Bayes’ Theorem

20. Bayesian Interpretation • Bayesian probability • experiments not repeatable, e.g. • what is the probability of another flood like 2008 Iowa flood? • what is the probability of observing 9/11 sized terrorist event? (11-35%, A. Clauset & R. Woodard, 2013) • measures degree of belief • Bayes’ Theorem: update our beliefs • e.g.

21. Bayesian Interpretation • e.g. I talked to a person yesterday, is he or she? 0.7 woman woman|long hair = 0.5 0.78 0.7 0.5 0.2 0.5 man woman

22. Bayes’ Theorem in Machine Learning Model Inference Parameters Data Data Likelihood Prior of model Posterior of model

23. Independence • two random variables are independent • Conditional Independence

24. A Question Are you going to win or not in along run? $2 $1 Red box Blue box You pay $1.5 to play the game

25. Today’s Topics • Basic Concepts • Two Rules of Probability • Bayes’ Theorem • Distributions and Statistics • Data Likelihood and MLE Pattern Recognition and Machine Learning Chapter 2

26. Expectation and Variance • function of a random variable • Expectation • Variance

27. Expectation You pay $1.5 to play the game. Are you going to win or not in a long run? $2 $1 Red box Blue box

28. Probability Distribution • Bernoulli Distribution: The outcome of an experiment can either be success (i.e., 1) and failure (i.e., 0).

29. Probability Distribution • Binomial Distribution: Random variable X stands for the number of times that experiments are successful.

30. Plots of Binomial Distribution

31. Probability Density • Continuous random variable probability density function (PDF)

32. Probability Density • Properties of PDF

33. Probability Distribution • Gaussian Distribution (Normal Distribution) Standard deviation

34. Probability Distribution • Multi-variate Gaussian Distribution

35. Today’s Topics • Basic Concepts • Two Rules of Probability • Bayes’ Theorem • Distributions and Statistics • Data Likelihood and MLE Pattern Recognition and Machine Learning Chapter 2

36. Data Likelihood • Observed Data: • Each generated from a distribution • Data Likelihood function • Under i.i.d (independent and identically distributed) assumption Generative Models

37. Maximum Likelihood Estimation • Observed Data: • Each generated from a distribution • Data Likelihood function • Under i.i.d (independent and identically distributed) assumption

38. Exercise: MLE • Observed Data: • Assume Bernoulli distribution • What is MLE of ?

39. Exercise: MLE • Observed Data: • Assume Gaussian Distribution • What is MLE of ?