1 / 24

Goodness of fit, confidence intervals and limits

fourth lecture. Goodness of fit, confidence intervals and limits. Jorge Andre Swieca School Campos do Jordão, January,2003. References. Statistical Data Analysis, G. Cowan , Oxford, 1998

ivana
Download Presentation

Goodness of fit, confidence intervals and limits

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. fourth lecture Goodness of fit, confidence intervals and limits Jorge Andre Swieca School Campos do Jordão, January,2003

  2. References • Statistical Data Analysis, G. Cowan, Oxford, 1998 • Statistics, A guide to the Use of Statistical Methods in the Physical Sciences, R. Barlow, J. Wiley & Sons, 1989; • Particle Data Group (PDG) Review of Particle Physics, 2002 electronic edition. • Data Analysis, Statistical and Computational Methods for Scientists and Engineers, S. Brandt, Third Edition, Springer, 1999

  3. Limits “Tens, como Hamlet, o pavor do desconhecido? Mas o que é conhecido? O que é que tu conheces, Para que chames desconhecido a qualquer coisa em especial?” Álvaro de Campos (Fernando Pessoa) “Se têm a verdade, guardem-na!” Lisbon Revisited, Álvaro de Campos

  4. Statistical tests How well the data stand in agreement with given predicted probabilities – hypothesis. null hypothesis H0 alternative function of measured variables: test statistics error first kind significance level error second kind power to discriminate against H1 power =

  5. Neyman-Pearson lemma H0 signal H1 background Where to place tcut? 1-D: efficiency (and purity) m-D: def. of acceptance region is not obvious Neyman-Pearson lemma: highest power (highest signal purity) for a given significance level α determined by the desired efficiency region of t-space such that

  6. 10 17 18 19 20 0 1 2 3 Goodness of fit how well a given null hypothesis H0 is compatible with the observed data (no reference to other alternative hypothesis) coins: N tosses, nh , nt= N - nh coin “fair’? H and T equal? binomial distribution, p=0.5 test statistic: nh N=20, nh=17 E[nh]=Np=10

  7. Goodness of fit P-value: probability P, under H0, obtain a result as compatible of less with H0 than the one actually observed. P=0.0026 P-value is a random variable, α is a constant specified before carrying out the test Bayesian statistics: use the Bayes theorem to assign a probability to H0 (specify the prior probability) P value is often interpreted incorrectly as a prob. to H0 P-value: fraction of times on would obtain data as compatible with H0 or less so if the experiment (20 coin tosses) were repeated under similar circunstances

  8. Goodness of fit “optional stopping problem” Easy to identify the region of values of t with equal or less degree of compatibility with the hypothesis than the observed value (alternate hypothesis: p ≠ 0.5)

  9. Significance of an observed signal Whether a discrepancy between data and expectation is sufficiently significant to merit a claim for a new discovery signal event ns, Poisson variable νS background event nb, Poisson variable νb prob. to observe n events: experiment: nobs events, quantify our degree of confidence in the discovery of a new effect (νS≠0) How likely is to find nobs events or more from background alone?

  10. Significance of an observed signal P(n>nobs)=1.7x10-4 Ex: expect νb=0.5, nobs= 5 this is not the prob. of the hypothesis νS=0 ! this is the prob., under the hypothesis νS=0, of obtaining as many events as observed or more.

  11. Significance of an observed signal How to report the measurement? estimate of ν : • only two std. deviations from zero • impression that νS is not very incompatible • with zero misleading: yes: prob. that a Poisson variable of mean νb will fluctuate up to nobs or higher no: prob. that a variable with mean nobs will fluctuate down to νb or lower

  12. Pearson’s test ni νi histogram of x with N bins construct a statistic which reflects the level of agreement between observed and expected histograms data aprox. gaussian, Poisson distributed with follow a distribution for N degrees of freedom • regardless of the distribution of x • distribution free larger larger discrepancy between data and the hypothesis

  13. Pearson’s test (rule of thumb for a good fit)

  14. Pearson’s test

  15. Pearson’s test Poisson variable with Before ni dist. as multinomial with prob. Set ntot = fixed Not testing the total number of expected and observed Events, but only the distribution of x. large number on entries in each bin pi known Follows a distribution for N-1 degrees of freedom In general, if m parameters estimated from data, nd = N - m

  16. Standard deviation as stat. error n observations of x, hypothesis p.d.f f(x;θ) ML: estimator for θ measurement analytic method RCF bound Monte Carlo graphical standard deviation repeated estimates each based on n obs.: estimator dist. centered around true value θ and with true estimated by and Most practical estimators: becomes approx. Gaussian in the large sample limit.

  17. prob. α prob.β Classical confidence intervals n obs. of x, evaluate an estimator for a param. θ obtained and its p.d.f. (for a given θ unknown)

  18. Classical confidence intervals prob. for estimator to be inside the belt regardless of θ monotonic incresing functions of θ

  19. a: hypothetical value of for which a fraction of the repeated estimt. would be higher than the obtain. Classical confidence intervals Usually: central confidence interval

  20. Classical confidence intervals Relationship between a conf. interval and a test of goodness of fit: test the hypothesys using having equal or less agreement than the result obtained P-value = α (random variable) and θ = a is specified Confidence interval: α is specified first, a is a random quantity depending on the data

  21. Classical confidence intervals Many experiments: the interval would include the true value in It does not mean that the probability that the true value of is in the fixed interval is Frequency interpretation: is not a random variable, but the interval fluctuates since it is constructed from data.

  22. Gaussian distributed Simple and very important application Central limit theorem: any estimator linear function of sum of random variables becomes Gaussian in the large sample limit. known, experiment resulted in

  23. Gaussian distributed

  24. Gaussian distributed Choose quantile 1 0.6827 1 0.8413 2 0.9544 2 0.9772 3 0.9973 3 0.9987 Choose confidence level 0.90 1.645 0.90 1.282 0.95 1.960 0.95 1.645 0.99 2.576 0.99 2.326.

More Related