1 / 19

Outline

Outline. Introduction to the Bayesian theory Bayesian Probability Bayes ’ Rule Bayesian Inference Historical Note Coin trials example Bayes rule and inference Elementary definition: Mahadevan . Probability, reliability, and statistical methods in engineering design. Wiley, 2000.

rey
Download Presentation

Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Outline • Introduction to the Bayesian theory • Bayesian Probability • Bayes’ Rule • Bayesian Inference • Historical Note • Coin trials example • Bayes rule and inference • Elementary definition: Mahadevan. Probability, reliability, and statistical methods in engineering design. Wiley, 2000. • Modern definition: Gelman, Andrew, et al. Bayesian data analysis. CRC press, 2003. • Example: Inference about a genetic probability

  2. Bayesian Probability • What is Bayesian probability ? • Classical (Frequentist) versus Bayesian (Alternative) definition • Classical: relative frequency of an event, given many repeated trials • Bayesian: degree of belief that it was true based on evidence at hand • Saturn mass estimation • Classical: mass is fixed but unknown. • Bayesian: mass in probabilistic way based on the observations, which represents our degree of belief. Data Analysis, A Bayesian Tutorial, DS Sivia, 1996. Always single value after many trials ? Not suitable to tackle this problem Classical approach Bayesian approach

  3. Bayes’ Rule • What is Bayes’ rule ? • Probability distribution (degree of belief) is obtained by combining prior knowledge (subjective) and observed data (objective). • Features • Integrated framework for uncertainty : aleatory uncertainty (classical statistics) & epistemic uncertainty (insufficient data, vague knowledge). • Bayesian updating: as more data provided, posterior PDF is used as prior at next, and values are updated to more confident information. Updated prior PDF Prior distribution Likelihood function Posterior distribution Observed data added Observed data

  4. Bayesian Inference • Deductive vs Inductive process • Deductive process: deduce outcome from hypothesis. • Inductive process: infer hypothesis(Cause) from outcome(observation). • In the Bayesian approach, inductive problem is easily solved by using the Bayes rule. Cause Induction (reverse):B & C observed. Thus, A is supported Cause Deduction (forward):If A then B. Observed data Observed data m=10, s=2 Observeddatax Given q Get data x. X ~ N(m,s) Estimate q. s Prior for q if available m

  5. Historical Note • Birth of Bayesian • Thomas Bayes (priest) proposed Bayes’ theory (1763):q of Binomial dist. is estimated using observed data.Laplace discovered, put his name (1812), generalized to many prob’s. • For more 100 years, Bayesian “degree of belief” was rejected due to vague and subjectivity. Objective “frequency” was accepted in statistics. • Jeffreys (1939) rediscovered, made modern theory (1961).Until 80s, still only a theory due to requirement for computation. • Flourishing of Bayesian • From year 1990, due to rapid growth of HW & SW, became practical. • Bayesian technique for practical use developed by math & statisticians, applied to areas of science (economics, medical) & engineering. , 1999

  6. Coin Trials Example • Problem • For a weighted (uneven) coin, probability of front side is to be determined based on the experiments. • Assume the true q is 0.78, obtained after ∞ trials.But we don’t know this. Only infer based on experiments. • Bayesian parameter estimation This is the parameter q to be estimated. Count of successes • Prior knowledge on p0(q ) • No prior information • Normal dist centered at 0.5 with s=0.05 • Uniform distribution [0.5, 0.7] Count of failures Outcome Posterior distribution of q Failure probability Given parameter Success probability Likelihood by Binomial dist. • Experiment data: x times out of n trials. • 4 out of 5 trials • 78 out of 100 trials

  7. Coin Trials Example • Estimated results • Case 1: no prior  4 out of 5 trials  78 out of 100 • Case 2: normal dist. N(0.5,0.05)  4 out of 5 trials  78 out of 100Due to wrong prior, only a great # trials will give true answer. • Case 3: unif. dist. [0.5 0.7]  4 out of 5 trials  78 out of 100can’t exceed the barrier due to the incorrect prior. • Followings are the degree of belief on “probability of front side q”as represented by PDF.

  8. Coin Trials Example • Posterior prediction • Once we have posterior PDF of q, we can predict future trials using this. • Let’s say we have obtained posterior PDF of q from 78 out of 100 trials.What is the predicted probability to get all 5 fronts when try 5 times ? If we know exact value 0.78, binom(5,5,0.78) = 0.785 = 0.289 Estimation process Posterior prediction process Monte Carlo simulation Draw random samples of q from PDF Compute p based on each q binom(5,5,q) Posterior PDF of q 10,000 samples of q 10,000 samples of predicted p

  9. Bayes rule and inference • Mahadevan. Probability, reliability, and statistical methods in engineering design. Wiley, 2000. • Axioms of probability • P(E): probability of an event E. • Let us consider two events E1, E2. • For mutually exclusive events, • When occurrence of a event precludes the other, mutually exclusive. • When joint occurrence E1∩E2 = E1E2 exists, • Conditional probability • When joint occurrence exists, • Symbol | denotes conditional probability or probability of event given the other has occurred. • For statistically independent events, • If mutually exclusive, we can’t have joint occurrence.

  10. Bayes rule and inference • Example 1 • E1: 80, E2: ≥90. P(E1∩E2) = ? P(E1UE2) = ? • Are E1, E2 mutually exclusive ? • E1: 80, 100. E2: ≥90. P(E1∩E2) = ? P(E1UE2) = ? • Are E1, E2 mutually exclusive ?

  11. Bayes rule and inference • Example 2 • P(F U E) = ? • Are F, E mutually exclusive ? Are F, E statistically independent ? • Example 3 • P(F U S) ? • Can we draw Venn diagram ? • P(F U S) when F, S are independent ? • Which probability is higher ?, i.e., more conservative ?

  12. Bayes rule and inference • Total probability (Forward problem) • Consider damage caused by 3 events: fire (F), wind (W), earthq (E). • Given P(F),P(W),P(E), then the probability of D: where P(D|Ei) is likelihood of D given Ei. • Bayes rule (Inverse problem) • Given D, then the probability of F: • This is evident from D F W E

  13. Bayes rule and inference • Example 4 • P(D) ? • P(F|D) ?

  14. Bayes rule and inference • Bayes rule for modern applications • Original is modified to • In Gelman, where often called marginal distribution of y.This represents sum of all possible values of q. • Interpretations • Prior probability: our state of knowledge about the truth of the hypothesis before we experience the data. • This belief is modified by the likelihoods of the experienced observations. • Then Posterior probability is obtained, which represents updated state of knowledge about the truth of the hypothesis. • In this sense, Bayes’ theorem gives us the process of learning.

  15. Bayes rule and inference • Posterior prediction • Make inference about an unknown observable. • Bayes rule was to make inference on q. • Posterior prediction is to make inference on another y, denoted by y ̃ conditional on the observed data y. • Interpretations • Posterior because it is conditional on the observed y • Predictive because it is prediction for a new observation y ̃

  16. Bayes rule and inference • Example 5 • P(F|D), P(W|D), P(E|D) without P(D) ?

  17. 1.4 Example: Inference about a genetic probability • Problem • Statement • Let Xg be normal gene whereas Xb is affected gene. • If two Xb’s found in woman, she is in disease. If one Xb found in man, he is in disease. • Let us estimate probability of Xbof a normal woman. • Observations • A woman’s father is normal  Xg Y; Brother is disease  Xb Y.Woman’s mother normal but  XgXb because of the brother. • She has two sons y1 & y2, neither of whom is affected. Husband is normal  Xg Y. • Prior knowledge • Regarding the Xb of the woman, we have no prior knowledge. • Bayesian inference • Unknown parameters and observations: • Introduce q for X such that if q =0, X=Xg, if q =1, X=Xb. • Introduce y for outcome such that if y=0, normal, if y=1, disease.Then the observations are y1=0 & y2=0.

  18. 1.4 Example: Inference about a genetic probability • Formulation • Prior distribution • Likelihood from observations • Posterior distribution • Estimation result • Probability that she doesn’t have bad gene is 80%. • Posterior PDF of q

  19. 1.4 Example: Inference about a genetic probability • Posterior prediction • What is the probability of disease if she has third son? • Answer is 0.01 • Bayesian updating by adding more data • Third son is found to be normal. How is q updated ? • In this case, previous posterior becomes now prior. • Observation is only one y. • Answer is 0.111

More Related