Bayesian Prior and Posterior

1. Bayesian Prior and Posterior Study Guide for ES205 Yu-Chi Ho Jonathan T. Lee Nov. 24, 2000

2. Outline • Conditional Density • Bayes Rule • Conjugate Distribution • Example • Other Conjugate Distributions • Application

3. N Conditional Density • The conditional probability density of w happening given x has occurred, assume px(x)  0:

4. N Bayes Rule • Replace the joint probability density function with the bottom equation from page 3:

5. N Conjugate Distribution • W: parameter of interest in some system • X: the independent and identical observation on the system • Since we know the model of the system, the conditional density of X|W could be easily computed, e.g.,

6. N Conjugate Distribution (cont.) • If the prior distribution of W belong to a family, for any size n and any values of the observations in the sample, the posterior distribution of W must also belong to the same family. This family is called a conjugate family of distributions.

7. N Example • An urn of white and red balls with unknown w being the fraction of the balls that are red. • Assume we can take n sample, X1, …, Xn, from the urn, with replacement, e.g, n i.i.d. samples.This is a Bernoulli distribution.

8. N Example (cont.) • Total number of red ball out of n trials, Y = X1 + … + Xn, has the binomial distribution • Assume the prior dist. of w is beta distribution with parameters  and 

9. Example (cont.) • The posterior distribution of W iswhich is also a beta distribution.

10. N Example (cont.) • Updating formula: • ’ =  + yPosterior (new) parameter =prior (old) parameter + # of red balls • ’ =  + (n – y) Posterior (new) parameter= prior (old) parameter + # of white balls

11. N Other Conjugate Distributions • The observations forms a Poisson distribution with an unknown value of the mean w. • The prior distribution of w is a gamma distribution with parameters  and . • The posterior is also a gamma distribution with parameters and  + n. • Updating formula:’ =  + y’ =  + n

12. N Other Conjugate Distributions (cont.) • The observations forms a negative binomial distribution with a specified r value and an unknown value of the mean w. • The prior distribution of w is a beta distribution with parameters  and . • The posterior is also a beta distribution with parameters  + rn and . • Updating formula:’ =  + rn’ =  + y

13. N Other Conjugate Distributions (cont.) • The observations forms a normal distribution with an unknown value of the mean w and specified precision r. • The prior distribution of w is a normal distribution with mean  and precision . • The posterior is also a normal distribution with mean and precision  + nr. • Updating formula:

14. N Other Conjugate Distributions (cont.) • The observations forms a normal distribution with the specified mean m and unknown precision w. • The prior distribution of w is a gamma distribution with parameters  and . • The posterior is also a gamma distribution with parameters and . • Updating formula:’ =  + n/2’ =  + ½

15. N Summary of the Conjugate Distributions

16. N Application • Estimate the state of the system based on the observations: Kalman filter.

17. References: • DeGroot, M. H., Optimal Statistical Decisions, McGraw-Hill, 1970. • Ho, Y.-C., Lecture Notes, Harvard University, 1997. • Larsen, R. J. and M. L. Marx, An Introduction to Mathematical Statistics and Its Applications, Prentice Hall, 1986.