- 144 Views
- Uploaded on
- Presentation posted in: General

Statistics for Particle Physics Lecture 5: Probability and Confidence

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

**1. **Statistics for Particle Physics Lecture 5: Probability and Confidence Roger Barlow
Manchester University
TAE, Santander,11 July 2006

**2. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 2 Probability Definition 1: Mathematical P(A) – the probability of some event A - is a number obeying the Kolmogorov axioms

**3. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 3 Probability Definition 2: Classical The probability P(A) is a property of an object that determines how often event A happens.
It is given by symmetry for equally-likely outcomes
Outcomes not equally-likely are reduced to equally-likely ones
Examples:
Tossing a coin:
P(H)=1/2
Throwing two dice
P(8)=5/36

**4. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 4 Cannot handle Bertrand’s Paradox A jug contains 1 glassful of water and between 1 and 2 glasses of wine
Q: What is the most probable wine:water ratio?
A: Between 1 and 2 ?3/2
Q: What is the most probable water:wine ratio?
A: Between 1/1 and 1/2 ?3/4
(3/2)?(3/4)-1

**5. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 5 Definition 3: Frequentist The probability P(A) is the limit (taken over some ensemble of as large number of cases)

**6. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 6 Problem (limitation) for the Frequentist definition P(A) depends on A and the ensemble
Eg: count 10 of a group of 30 with beards.
P(beard)=1/3

**7. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 7 Big problem for the Frequentist definition Cannot be applied to unique events
‘It will probably rain tomorrow’
Is unscientific
`The statement
“It will rain tomorrow”
is probably true.’
Is quite OK

**8. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 8 Interpreting physics results: Mt =173±2 GeV/c2 Can’t say
‘Mt has a 68% probability of lying between 171 and 175 GeV/c2’
Have to say
‘The statement “Mt lies between 171 and 175 GeV/c2”has a 68% probability of being true’
i.e. if you always say a value lies within its error bars, you will be right 68% of the time.
Say “Mt lies between 171 and 175 GeV/c2” with 68%
Confidence. Or 169-177 with 95% confidence. Or…

**9. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 9 Definition 4: Subjective (Bayesian) P(A) is your degree of belief in A;
You will accept a bet on A if the odds are better than
1-P to P
A can be Anything :
Beards, Rain, particle decays, conjectures, theories
But two different people can have different values for P(A)

**10. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 10 Bayes Theorem Often used for subjective probability Conditional Probability P(A|B)
P(A & B)= P(B) P(A|B)
P(A & B)= P(A) P(B|A)
Example:
W=white jacket B=bald
P(W&B)=(2/4)x(1/2)
or (1/4)x(1/1)
P(W|B)
= 1 x (1/4) =1/2
(2/4)

**11. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 11 Bayes’ Theorem and subjective probability

**12. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 12 Frequentist versus Bayesian? Statisticians do a lot of work with Bayesian statistics and there are a lot of useful ideas. But they are careful about checking for robustness under choice of prior.
Beware snake-oil merchants in the physics community who will sell you Bayesian statistics (new – cool – easy – intuitive) and don’t bother about robustness.
Use Frequentist methods when you can and Bayesian when you can’t (and check for robustness.) But ALWAYS be aware which you are using.

**13. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 13 Confidence Levels: The Straightforward Example

**14. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 14 Confidence Levels Can quote at any level
(68%, 95%, 99%…)
Upper or lower or twosided
(x<U x<L L<x<U)
Two-sided has further choice
(central, shortest…)

**15. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 15 The Frequentist Twist

**16. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 16 Confidence belt (Neyman interval) Observe n events. What can you say about ? ?
Use e-??n/n!
For any true ? the probability that (n, ?) is within the belt is 68% (or more) by construction
For any n, ? lies in [?-, ?+] at 68% confidence

**17. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 17 Discrete Distributions

**18. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 18 Coverage “L<x<U” @ 95% confidence
(or “x>L” or “x<U”)
This statement belongs to an ensemble of similar statements of which at least* 95% are true
95% is the coverage
*Maybe more. Overcoverage
EG composite hypotheses.

**19. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 19 Maximum Likelihood and Confidence Levels ML estimator (large N) has variance given by MVB
At peak For large N
Ln L is a parabola (L is a Gaussian)

**20. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 20 2(+) parameters Fix b, find 68% confidence range for a, using ?ln L=-˝
Fix a, find 68% range for b
Combination (square) has 0.682=46%

**21. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 21 What happens if there’s nothing there? Even if your analysis finds no events, this is still useful information about the way the universe is built
Want to say more than: “We looked for X, we didn’t see it.”
Need statistics – which can’t prove anything.
“We show that X probably has a mass greater than../a coupling smaller than…”

**22. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 22 Interpreting null result Your analysis searches for events. Sees none.
Use Poisson formula: P(n; ?)=e-??n/n!
Small ? could well give 0 events
? =0.5 gives P(0)=61%
? =1.0 gives P(0)=37%
? =2.3 gives P(0)=10%
? =3.0 gives P(0)=5%
If you always say ‘? ?3.0’ you will be right (at least) 95% of the time. ? ?3.0 – with 95% confidence (a.k.a 5% significance.)
‘If ? is actually 3, or more, the probability of a fluctuation as far as zero is only 5%, or less.’
? given by model parameters. Limit on ? translates to limit on mass, coupling, ,branching ratio or whatever

**23. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 23 Problems for Frequentists Weigh
object+container with some Gaussian precision
Get reading R
R-? < M+C < R+? @68%
R-C-? < M < R-C+? @68%
E.g. C=50, R=141, ?=10
81<M<101 @68%
E.g. C=50, R=55, ?=10
-5 < M < 5 @68%
E.g. C=50, R=31, ?=10
-29 < M < -9 @68%

**24. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 24 Bayes to the rescue Standard (Gaussian) measurement
No prior knowledge of true value
No prior knowledge of measurement result
P(Data|Theory) is Gaussian
P(Theory|Data) is Gaussian
Interpret this with Probability statements in any way you please

**25. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 25 Bayesian Confidence Intervals (contd) Observe (say) 2 events
P(?;2)?P(2; ?)=e-? ?2
Normalise and interpret
If you know background mean is 1.7, then you know ?>1.7
Multiply, normalise and interpret

**26. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 26 Bayesian Strategy Prior is uniform for positive ?, zero for negative ?. No problem.
Get requirement (for n observed, known background b, 90% upper limit)
0.1=??nexp(-?+-b) (?++b)r/r!
??nexp(-b) br/r!
Known as “the old PDG formula” or “Helene’s formula” or “that heap of crap”

**27. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 27 Bayes at work

**28. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 28 Bayes at work again

**29. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 29 ‘Objective Prior’ (Jeffreys) Transform to a variable q(M) for which the Fisher information is constant
For a location parameter with P(x;M)=f(x+M) use M
For scale parameter with P(x;M)=Mf(x) use ln M
For a Poisson l use prior 1/??
For a Binomial with probability p use prior 1/?p(1-p)
This has never really caught on

**30. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 30 Feldman-Cousins Unified Method Physicists are human
Ideal Physicist
Choose Strategy
Examine data
Quote result
Real Physicist
Examine data
Choose Strategy
Quote Result

**31. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 31 “Flip-Flopping”

**32. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 32 Solution: Construct belt that does the flip-flopping

**33. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 33 Better Strategy N is Poisson from S+B
B known, may be large
E.g. B=9.2,S=0 and N=1
P=.1% - not in C-G band
But any S>0 will be worse To construct band for a given S:
For all N:
Find P(N;S+B) and
Pbest=P(N;N) if (N>B) else P(N;B)
Rank on P/Pbest
Accept N into band until ? P(N;S+B) ?90%

**34. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 34 Feldman and Cousins Summary Makes us more honest (a bit)
Avoids forbidden regions in a Frequentist way
Not easy to calculate
Has to be done separately for each value of B
Can lead to 2-tailed limits where you don’t want to claim a discovery
Weird effects for N=0; larger B gives lower (=better) upper limit

**35. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 35 Another Strategy CLs As used by LEP Higgs working group
Generalisation of Helene formula
Some quantity Q. Could be number of events, or something more clever
CLb=P(Q or less|b) CLs+b=P(Q or less|s+b)
CLs=CLs+b/CLb
Used as confidence level. Optimise strategy using it and quote results

**36. **R Barlow TAE 2006 Statistics in Particle Physics - Lecture 5 36 Conclusion 4 ways to define Probability
Mathematical
Classical
Frequentist
Subjective
Each has strong points and weak points.
Be prepared to understand and use them all
Confidence levels –
Zero events = 95% CL upper limit of 3 events
Be suspicious of anything you don’t understand. The authors probably don’t understand it either.
Say what you’re doing.
If you’re integrating a likelihood you are a Bayesian. I hope you know what you’re doing.
The End