Methods of Experimental Particle Physics

1. Methods of Experimental Particle Physics Alexei Safonov Lecture #24

2. The Likelihood Function • Example from last time: • Where • If S+B: • Assume signal mass is known • Such L is a function of: • L(Ni, S, B) • It depends both on data and the model parameters • Once the measurement is done, Ni is fixed • In Bayesian approach you stop here: “whatever happened in data has happened, what can I now tell about the parameters” • Frequentists will want to spend more time on this as you want to measure frequencies of various outcomes as those are your final probabilities

3. Parameter Extraction • The likelihood function discussed earlier is the basis for all further interpretations • While the absolute value of L is not very telling on its own (need to calculate the p-value), relative changes in L are • If you have a choice of parameters for the hypothesis, you can check how L is changing as a function of that parameter (fix “data” to what you saw in the data) • Typical example: • Likelihood function versus normalization of the signal S • Some values lead to better agreement with the data than others

4. Bayesian Interpretation • While the easiest way to pass all the information about your measurement is via the Likelihood function, one usually want a simple way to quantify the results • Conventionally, people report 68% C.L. intervals for the parameters being • Many ways to choose 68%, need a convention • One example is highest posterior density intervals • “Probability outside is less than on the inside” • C.L. stands for: • Confidence Level when using frequentist approach • Credibility Level is Bayesian

5. Frequentist Limits • Suppose that the true value of qis q0: • q0lies between q1 (x) and q22(x) if and only if x lies between x1(q0) and x2(q0) • The two events have the same probability, • Since it is true for any value q0 • This is Neyman procedure. Constructed to ensure the desired coverage • Basically coverage is the probability that the true parameter is within the interval you report • If you say that at 68% C.L. z=1+/-0.1, it better be that in 68% of experiments the true value of z is within the interval you report

6. Setting Frequentist Limits • Once you constructed the Likelihood, you can do pseudoexperiments • Find what the limits should be to ensure desired coverage for any outcome x • Report the limits for the x you saw in your data • You still need how to define the borders of the confidence belt • Also a convention, one option is require a/2 on each side • Or can follow a different procedure (e.g. Feldman-Cousins)

7. Multi-Parameter Likelihoods • Not too difficult to extend what we did onto a multi-dimensional space: • Your reported intervals will become ellipses: • The conventional way to report the numeric results “on paper” is to “overcover” by reporting rectangular box limits • But you can also report the correlation matrix or make a 2D-plot like above if prudent, happens often

8. One Sided Limits • It is typical in HEP to look for things that are at the edge of your sensitivity • You frequently can’t “see” the signal, but you still want to say that the cross-section for that new process can’t be larger than X • Also very useful information for theorists and model builders as your results can rule out a whole bunch of models or parameter space of the models • Can do it for either Bayesian or Frequentist methods • Most of the time fairly straightforward – either construct a one-sided intervals with known coverage or calculate the integral from 0 to x in Bayesian case

9. Bayesian vsFrequentist • Practical differences usually only really relevant when the measured parameter is near a physical boundary • Example: you looking for signal by measuring the number of some type of events during one year. You estimated that the background processes should yield B=100 events on average, so any “excess” in data (D=S+B) would indicate a potential signal • If you observe 300 events, you clearly see signal and can easily do the interpretation • If you see 100 events, you are still okay • But what if you see 30 events or even zero events? • What would be the upper 95% C.L. limit for signal in each case? Does that make sense?

10. Bayesian vsFrequentist • Observing 0 events when you expect 100: • Frequentist limit: • If you don’t attempt to limit the outcomes to the positive values of S, you will likely get a negative S • If you try to account for it, still when you construct the belt, smaller values of S+B will be giving an increased probability to observe 0 events, so the limit will depend on B • Bayesian limit: • For N=0: • Once you say S can only be positive, you will be calculating: • The limit is independent of B! • What does it mean?

11. Practical Notes • You usually will calculate –log(L) instead of L • Assuming you are doing a measurement, the minimum of –log(L) is the maximum of L, so that gives the most likely parameter value • Changing – log(L) by +/-1/2 (think of taking a log of a gaussian distribution – it will give you (x-x0)2/2s2 so x shifted by sigma gives ½) gives you 1 sigma deviation in the parameter (68% C.L.) • With the MLS method, you vary it by 1 instead: • MINUIT is the most used minimization package in HEP (it is part of ROOT), easy to use in simple cases, some experience required for more complex ones

12. Some good to remember numbers • Upper 90% and 95% limit on the rate of a Poisson process in the absence of background when you observe n events • If you observe no events, the limit is 3 events • In Bayesian case, this would also be true for any expected B rate

13. Un-Binned Likelihood • Binned likelihood is easy to interpret and gives consistent and predictable outcomes • But there is a choice of bin size and strictly speaking you are loosing information by clamping things together • Can you avoid that? • Decrease the size of the bins, in the extreme limit – unbinned likelihood: • ) • where nu is the function value at the point where the data point happened • Is there a problem with this?

14. Next Time • Take as a use case Higgs results from CMS and walk through various plots • Upper limits, various definitions of tests, central values, confidence intervals etc