# Statistics for Particle Physics Lecture 5: Probability and Confidence PowerPoint PPT Presentation

Statistics for Particle Physics Lecture 5: Probability and Confidence

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

#### Presentation Transcript

1. Statistics for Particle Physics Lecture 5: Probability and Confidence Roger Barlow Manchester University TAE, Santander,11 July 2006

2. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 2 Probability Definition 1: Mathematical P(A) – the probability of some event A - is a number obeying the Kolmogorov axioms

3. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 3 Probability Definition 2: Classical The probability P(A) is a property of an object that determines how often event A happens. It is given by symmetry for equally-likely outcomes Outcomes not equally-likely are reduced to equally-likely ones Examples: Tossing a coin: P(H)=1/2 Throwing two dice P(8)=5/36

4. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 4 Cannot handle Bertrand’s Paradox A jug contains 1 glassful of water and between 1 and 2 glasses of wine Q: What is the most probable wine:water ratio? A: Between 1 and 2 ?3/2 Q: What is the most probable water:wine ratio? A: Between 1/1 and 1/2 ?3/4 (3/2)?(3/4)-1

5. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 5 Definition 3: Frequentist The probability P(A) is the limit (taken over some ensemble of as large number of cases)

6. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 6 Problem (limitation) for the Frequentist definition P(A) depends on A and the ensemble Eg: count 10 of a group of 30 with beards. P(beard)=1/3

7. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 7 Big problem for the Frequentist definition Cannot be applied to unique events ‘It will probably rain tomorrow’ Is unscientific `The statement “It will rain tomorrow” is probably true.’ Is quite OK

8. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 8 Interpreting physics results: Mt =173±2 GeV/c2 Can’t say ‘Mt has a 68% probability of lying between 171 and 175 GeV/c2’ Have to say ‘The statement “Mt lies between 171 and 175 GeV/c2”has a 68% probability of being true’ i.e. if you always say a value lies within its error bars, you will be right 68% of the time. Say “Mt lies between 171 and 175 GeV/c2” with 68% Confidence. Or 169-177 with 95% confidence. Or…

9. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 9 Definition 4: Subjective (Bayesian) P(A) is your degree of belief in A; You will accept a bet on A if the odds are better than 1-P to P A can be Anything : Beards, Rain, particle decays, conjectures, theories But two different people can have different values for P(A)

10. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 10 Bayes Theorem Often used for subjective probability Conditional Probability P(A|B) P(A & B)= P(B) P(A|B) P(A & B)= P(A) P(B|A) Example: W=white jacket B=bald P(W&B)=(2/4)x(1/2) or (1/4)x(1/1) P(W|B) = 1 x (1/4) =1/2 (2/4)

11. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 11 Bayes’ Theorem and subjective probability

12. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 12 Frequentist versus Bayesian? Statisticians do a lot of work with Bayesian statistics and there are a lot of useful ideas. But they are careful about checking for robustness under choice of prior. Beware snake-oil merchants in the physics community who will sell you Bayesian statistics (new – cool – easy – intuitive) and don’t bother about robustness. Use Frequentist methods when you can and Bayesian when you can’t (and check for robustness.) But ALWAYS be aware which you are using.

13. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 13 Confidence Levels: The Straightforward Example

14. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 14 Confidence Levels Can quote at any level (68%, 95%, 99%…) Upper or lower or twosided (x<U x<L L<x<U) Two-sided has further choice (central, shortest…)

15. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 15 The Frequentist Twist

16. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 16 Confidence belt (Neyman interval) Observe n events. What can you say about ? ? Use e-??n/n! For any true ? the probability that (n, ?) is within the belt is 68% (or more) by construction For any n, ? lies in [?-, ?+] at 68% confidence

17. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 17 Discrete Distributions

18. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 18 Coverage “L<x<U” @ 95% confidence (or “x>L” or “x<U”) This statement belongs to an ensemble of similar statements of which at least* 95% are true 95% is the coverage *Maybe more. Overcoverage EG composite hypotheses.

19. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 19 Maximum Likelihood and Confidence Levels ML estimator (large N) has variance given by MVB At peak For large N Ln L is a parabola (L is a Gaussian)

20. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 20 2(+) parameters Fix b, find 68% confidence range for a, using ?ln L=-˝ Fix a, find 68% range for b Combination (square) has 0.682=46%

21. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 21 What happens if there’s nothing there? Even if your analysis finds no events, this is still useful information about the way the universe is built Want to say more than: “We looked for X, we didn’t see it.” Need statistics – which can’t prove anything. “We show that X probably has a mass greater than../a coupling smaller than…”

22. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 22 Interpreting null result Your analysis searches for events. Sees none. Use Poisson formula: P(n; ?)=e-??n/n! Small ? could well give 0 events ? =0.5 gives P(0)=61% ? =1.0 gives P(0)=37% ? =2.3 gives P(0)=10% ? =3.0 gives P(0)=5% If you always say ‘? ?3.0’ you will be right (at least) 95% of the time. ? ?3.0 – with 95% confidence (a.k.a 5% significance.) ‘If ? is actually 3, or more, the probability of a fluctuation as far as zero is only 5%, or less.’ ? given by model parameters. Limit on ? translates to limit on mass, coupling, ,branching ratio or whatever

23. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 23 Problems for Frequentists Weigh object+container with some Gaussian precision Get reading R R-? < M+C < R+? @68% R-C-? < M < R-C+? @68% E.g. C=50, R=141, ?=10 81<M<101 @68% E.g. C=50, R=55, ?=10 -5 < M < 5 @68% E.g. C=50, R=31, ?=10 -29 < M < -9 @68%

24. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 24 Bayes to the rescue Standard (Gaussian) measurement No prior knowledge of true value No prior knowledge of measurement result P(Data|Theory) is Gaussian P(Theory|Data) is Gaussian Interpret this with Probability statements in any way you please

25. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 25 Bayesian Confidence Intervals (contd) Observe (say) 2 events P(?;2)?P(2; ?)=e-? ?2 Normalise and interpret If you know background mean is 1.7, then you know ?>1.7 Multiply, normalise and interpret

26. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 26 Bayesian Strategy Prior is uniform for positive ?, zero for negative ?. No problem. Get requirement (for n observed, known background b, 90% upper limit) 0.1=??nexp(-?+-b) (?++b)r/r! ??nexp(-b) br/r! Known as “the old PDG formula” or “Helene’s formula” or “that heap of crap”

27. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 27 Bayes at work

28. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 28 Bayes at work again

29. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 29 ‘Objective Prior’ (Jeffreys) Transform to a variable q(M) for which the Fisher information is constant For a location parameter with P(x;M)=f(x+M) use M For scale parameter with P(x;M)=Mf(x) use ln M For a Poisson l use prior 1/?? For a Binomial with probability p use prior 1/?p(1-p) This has never really caught on

30. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 30 Feldman-Cousins Unified Method Physicists are human Ideal Physicist Choose Strategy Examine data Quote result Real Physicist Examine data Choose Strategy Quote Result

31. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 31 “Flip-Flopping”

32. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 32 Solution: Construct belt that does the flip-flopping

33. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 33 Better Strategy N is Poisson from S+B B known, may be large E.g. B=9.2,S=0 and N=1 P=.1% - not in C-G band But any S>0 will be worse To construct band for a given S: For all N: Find P(N;S+B) and Pbest=P(N;N) if (N>B) else P(N;B) Rank on P/Pbest Accept N into band until ? P(N;S+B) ?90%

34. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 34 Feldman and Cousins Summary Makes us more honest (a bit) Avoids forbidden regions in a Frequentist way Not easy to calculate Has to be done separately for each value of B Can lead to 2-tailed limits where you don’t want to claim a discovery Weird effects for N=0; larger B gives lower (=better) upper limit

35. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 35 Another Strategy CLs As used by LEP Higgs working group Generalisation of Helene formula Some quantity Q. Could be number of events, or something more clever CLb=P(Q or less|b) CLs+b=P(Q or less|s+b) CLs=CLs+b/CLb Used as confidence level. Optimise strategy using it and quote results

36. R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 36 Conclusion 4 ways to define Probability Mathematical Classical Frequentist Subjective Each has strong points and weak points. Be prepared to understand and use them all Confidence levels – Zero events = 95% CL upper limit of 3 events Be suspicious of anything you don’t understand. The authors probably don’t understand it either. Say what you’re doing. If you’re integrating a likelihood you are a Bayesian. I hope you know what you’re doing. The End