statistics in hep 2 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Statistics In HEP 2 PowerPoint Presentation
Download Presentation
Statistics In HEP 2

Loading in 2 Seconds...

play fullscreen
1 / 22

Statistics In HEP 2 - PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on

Statistics In HEP 2. How do we understand/interpret our measurements. Outline. Maximum Likelihood fit strict frequentist Neyman – confidence intervals what “bothers” people with them Feldmans /Cousins confidence belts/intervals Bayesian treatement of ‘unphysical’ results

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Statistics In HEP 2' - nhung


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
statistics in hep 2
Statistics In HEP 2

How do we understand/interpret our measurements

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

outline
Outline
  • Maximum Likelihood fit
  • strict frequentist Neyman – confidence intervals
    • what “bothers” people with them
  • Feldmans/Cousins confidence belts/intervals
  • Bayesian treatement of ‘unphysical’ results
  • How the LEP-Higgs limit was derived
  • what about systematic uncertainties?
    • Profile Likleihood

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

parameter estimation
Parameter Estimation
  • want to measure/estimate some parameter
    • e.g. mass, polarisation, etc..
  • observe:
    • e.gobservables for events
  • “hypothesis” i.e. PDF ) - distribution for given
    • e.g. diff. cross section
    •  independent events: P(,..
  • for fixedregard ) as function of (i.e. Likelihood! L())
    • close to Likelihood L() will be large
  • try to maximuseL()
  • typically:
    • minimize -2Log(L()) 

 Maximum Likelihood estimator

Glen Cowan: Statistical data analysis

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

maximum likelihood estimator
Maximum Likelihood Estimator
  • example: PDF(x) = Gauss(x,μ,σ) 
  • estimator for from the data measured in an experiment
  • full Likelihood
  • typically: Note: It’s a function of !

For Gaussian PDFs:

==

  • from interval
  •  variance on the estimate

=1

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

parameter estimation1
Parameter Estimation

properties of estimators

  • biased or unbiased
  • large or small variance
  •  distribution of on many measurements ?
  • Small bias and small variance are typically “in conflict”
  • Maximum Likelihood is typically unbiased only in the limit
  • If Likelihood function is “Gaussian” (often the case for large K central limit theorem)
  • get “error” estimate from or -2
  • if (very) none Gaussian
    •  revert typically to (classical) Neyman confidence intervals

Glen Cowan:

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

classical confidence intervals
Classical Confidence Intervals
  • another way to look at a measurement rigorously “frequentist”
  • Neymans Confidence belt for CL α (e.g. 90%)
  • each μhypotheticallytrue has a PDF of how the measured values will be distributed
  • determine the (central) intervals (“acceptance region”) in these PDFs such that they contain α
  • do this for ALL μhyp.true
  • connect all the “red dots”  confidence belt
  • measure xobs :
  •  conf. interval =[μ1, μ2] given by vertical line intersecting the belt.

μhypothetically true

μ2

μ1

  • by construction: for each xmeas. (taken according PDF() the confidence interval [μ1, μ2] contains in α = 90% cases

Feldman/Cousin (1998)

xmeasured

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

classical confidence intervals1
Classical Confidence Intervals
  • another way to look at a measurement rigorously “frequentist”
  • Neymans Confidence belt for CL α (e.g. 90%)
  • conf.interval =[μ1, μ2] given by
  • verticalline intersecting the belt.
  • by construction:
  • if the true value were
  • lies in [,] if it intersects ▐
  • xmeasintersects ▬ as in 90% (that’s how it was constructed)
  • only those xmeasgive [,]’s that intersect with the ▬
  • 90% of intervals cover

μhypothetically true

μ2

μ1

  • is Gaussian ( central 68% Neyman Conf. Intervals Max. Likelihood + its “error” estimate ;

Feldman/Cousin(1998):

xmeasured

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

flip flop
Flip-Flop
  • When to quote measuremt or a limit!
    • estimate Gaussian distributed quantity that cannot be < 0 (e.g. mass)
    • same Neuman confidence belt construction as before:
      • once for measurement (two sided, each tail contains 5% )
      • once for limit (one sided tails contains 10%)
  • decide: if <0 assume you =0
    • conservative
  • if you observe <3
    • quote upper limit only
  • if you observe >3
    • quote a measurement
  • induces “undercovering” as this acceptance region contains only 85% !!

Feldman/Cousins(1998)

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

some things people don t like
Some things people don’t like..
  • same example:
  • estimate Gaussian distributed quantitythat cannot be < 0 (e.g. mass)
  • using proper confidence belt
  • assume:
    •  confidence interval is EMPTY!
  • Note: that’s OK from the frequentist interpretation
  • in 90% of (hypothetical) measurements.
  • Obviously we were ‘unlucky’ to pick one out of the remaining 10%

Feldman/Cousins(1998).

  • nontheless: tempted to “flip-flop” ??? tsz .. tsz.. tsz..

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

feldman cousins a unified approach
Feldman Cousins: a Unified Approach
  • How we determine the “acceptance” region for each μhyp.true is up to us as long as it covers the desired integral of size α (e.g. 90%)
  • include those “xmeas. ” for which the large likelihood ratio first:
  • here: either the observation or the closes ALLOWED
  • α =90%

No “empty intervals anymore!

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

being lucky
Being Lucky…
  • give upper limit on signal on top of know (mean) background
    •  n=s+b from a possion distribution
  • Neyman: draw confidence belt with
    • “” in the “y-axis” (the possible true values of )
  • 2 experiments (E1 and E2)
  • ,
  • observing (out of luck) 0
  • E1: 95% limit on
  • E2: 95% limit on
  •  UNFAIR ! ?

sorry… the plots don’t match: one is of 90%CL the other for 95%CL

observed n

Glen Cowan: Statistical data analysis

for fixed Background

Background

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

being lucky1
Being Lucky …
  • Feldman/Cousins confidence belts
    • motivated by “popular” ‘Bayesian’ approaches to handle such problems.
  • Bayesian: rather than constructing Confidence belts:
  • turn Likelihood for (on given )into Posterior probability on
  • add prior probability on “s”:
  • Feldman/Cousins
  • there is still SOME “unfairness”
  • perfectly “fine” in frequentist interpretation:
  • should quote “limit+sensitivity”

0 1 10

Upper limit on signal

Background b

Helene(1983).

Feldman/Cousins(1998).

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

statistical tests in particle searches
Statistical Tests in Particle Searches
  • exclusion limits
  • upper limit on cross section (lower limit on mass scale)
  • (σ < limit as otherwise we would have seen it)
  • need to estimate probability of downward fluctuation of s+b
  • try to “disprove” H0 = s+b
  • better: find minimal s, for which you can sill excludeH0 = s+ba pre-specified Confidence Level

discoveries

need to estimate probability of upward fluctuation of b

try to disprove H0 =“background only”

type 1 error

b only

Nobs for 5s discovery

P = 2.8 10–7

  • b only
  • s+b

possible observation

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

which test statistic to used
Which Test Statistic to used?
  • P(t_
  • exclusion limit:
  • test statistic does not necessarily have to be simply the counted number of events:
    • remember Neyman Pearson  Likelihood ratio
  • pre-specify  95% CL on exclusion
  • make your measurements
    • if accepted (i.e. in “chritical (red) region”)
    • where you decided to “recjet” H0 = s+b
    • (i.e. what would have been the chance for THIS particular measurement to still have been “fooled” and there would have actually BEEN a signal)
  • s+b
  • b only

t

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

example lep higgs search
Example :LEP-Higgs search

Remember: there were 4 experiments, many different search channels

 treat different exerpiments just like “more channels”

  • Evaluate how the -2lnQ is distributed for
    • background only
    • signal+background
    • (note: needs to be done for all Higgs masses)
  • example: mH=115GeV/c2
  • more signal like
  • more background like

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

example lep sm higgs limit
Example: LEP SM Higgs Limit

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

example lep higgs search1
Example LEP Higgs Search
  • In order to “avoid” the possible “problem” of Being Lucky when setting the limit
  • rather than “quoting” in addition the expected sensitivity
  • weight your CLs+b by it:
  • more signal like
  • more background like

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

systmatic uncertainties
Systmatic Uncertainties
  • standard popular way: (Cousin/Highland)
    • integrate over all systematic errors and their “Probability distribution)
    •  marginalisation of the “joint probability density of measurement paremters and systematic error)
    • “hybrid” frequentist intervals and Bayesian systematic
    • has been shown to have possible large “undercoverage” for very small p-values /large significances (i.e. underestimate the chance of “false discovery” !!

! Bayesian ! (probability of the systematic parameter)

  • LEP-Higgs: generaged MC to get the PDFs with “varying” param. with systematic uncertainty
  •  essentiall the same as “integrating over”  need probability density for “how these parameters vary”

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

systematic uncertainties
Systematic Uncertainties
  • Why don’t we:
    • include any systematic uncertainly as “free parameter” in the fit
  • eg. measure background contribution under signal peak in sidebands
  • measurement + extrapolation into side bands have uncertainty
  • but you can parametrise your expected background such that:
  •  if sideband measurement gives this data  then b=…

Note: no need to specify prior probability

  • Build your Likelyhood function such that it includes:
    • your parameters of interest
    • those describing the influcene of the sys. uncertainty
    •  nuisance parameters

K.Cranmer Phystat2003

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

nuisance parameters and profile liklihood
Nuisance Parameters and Profile Liklihood
  • Build your Likelyhood function such that it includes:
    • your parameters of interest
    • those describing the influcene of the sys. uncertainty
    •  nuisance parameters

“ratio of likelihoods”, why ?

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

profile likelihood
Profile Likelihood

“ratio of likelihoods”, why ?

  • Why not simply using L(m,q) as test statistics ?
  • The number of degrees of freedom of the fit would be Nq+1m
  • However, we are not interested in the values of q( they are nuisance !)
  • Additional degrees of freedom dilute interesting information on m
  • The “profile likelihood” (= ratio of maximum likelihoods) concentrates the information on what we are interested in
  • It is just as we usually do for chi-squared: Dc2(m) = c2(m,qbest’) – c2(mbest, qbest)
  • Nd.o.f. of Dc2(m) is 1, and value of c2(mbest, qbest) measures “Goodness-of-fit”

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP

summary
Summary
  • Maximum Likelihood fit to estimate paremters
  • what to do if estimator is non-gaussion:
    • Neyman– confidence intervals
    • what “bothers” people with them
  • Feldmans/Cousins confidene belts/intervals
    • unifies “limit” or “measurement” confidence belts
  • CLs … the HEP limit;
    • CLs … ratio of “p-values” … statisticians don’t like that
    • new idea: Power Constrained limits
      • rather than specifying “sensitivity” and “neymand conf. interval”
      • decide beforehand that you’ll “accept” limits only if the where your exerpiment has sufficient “power” i.e. “sensitivity !
  • .. a bit about Profile Likelihood, systematic error.

Hadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP