**Bayesian Prior and Posterior** Study Guide for ES205 Yu-Chi Ho Jonathan T. Lee Nov. 24, 2000

**Outline** • Conditional Density • Bayes Rule • Conjugate Distribution • Example • Other Conjugate Distributions • Application

**N** Conditional Density • The conditional probability density of w happening given x has occurred, assume px(x) 0:

**N** Bayes Rule • Replace the joint probability density function with the bottom equation from page 3:

**N** Conjugate Distribution • W: parameter of interest in some system • X: the independent and identical observation on the system • Since we know the model of the system, the conditional density of X|W could be easily computed, e.g.,

**N** Conjugate Distribution (cont.) • If the prior distribution of W belong to a family, for any size n and any values of the observations in the sample, the posterior distribution of W must also belong to the same family. This family is called a conjugate family of distributions.

**N** Example • An urn of white and red balls with unknown w being the fraction of the balls that are red. • Assume we can take n sample, X1, …, Xn, from the urn, with replacement, e.g, n i.i.d. samples.This is a Bernoulli distribution.

**N** Example (cont.) • Total number of red ball out of n trials, Y = X1 + … + Xn, has the binomial distribution • Assume the prior dist. of w is beta distribution with parameters and

**Example (cont.)** • The posterior distribution of W iswhich is also a beta distribution.

**N** Example (cont.) • Updating formula: • ’ = + yPosterior (new) parameter =prior (old) parameter + # of red balls • ’ = + (n – y) Posterior (new) parameter= prior (old) parameter + # of white balls

**N** Other Conjugate Distributions • The observations forms a Poisson distribution with an unknown value of the mean w. • The prior distribution of w is a gamma distribution with parameters and . • The posterior is also a gamma distribution with parameters and + n. • Updating formula:’ = + y’ = + n

**N** Other Conjugate Distributions (cont.) • The observations forms a negative binomial distribution with a specified r value and an unknown value of the mean w. • The prior distribution of w is a beta distribution with parameters and . • The posterior is also a beta distribution with parameters + rn and . • Updating formula:’ = + rn’ = + y

**N** Other Conjugate Distributions (cont.) • The observations forms a normal distribution with an unknown value of the mean w and specified precision r. • The prior distribution of w is a normal distribution with mean and precision . • The posterior is also a normal distribution with mean and precision + nr. • Updating formula:

**N** Other Conjugate Distributions (cont.) • The observations forms a normal distribution with the specified mean m and unknown precision w. • The prior distribution of w is a gamma distribution with parameters and . • The posterior is also a gamma distribution with parameters and . • Updating formula:’ = + n/2’ = + ½

**N** Summary of the Conjugate Distributions

**N** Application • Estimate the state of the system based on the observations: Kalman filter.

**References:** • DeGroot, M. H., Optimal Statistical Decisions, McGraw-Hill, 1970. • Ho, Y.-C., Lecture Notes, Harvard University, 1997. • Larsen, R. J. and M. L. Marx, An Introduction to Mathematical Statistics and Its Applications, Prentice Hall, 1986.