Discussion on significance
1 / 14

Discussion on significance - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Discussion on significance. ATLAS Statistics Forum CERN/Phone, 2 December, 2009. Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan. p -values. The standard way to quantify the significance of a discovery

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Discussion on significance

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Discussion on significance

ATLAS Statistics Forum

CERN/Phone, 2 December, 2009

Glen Cowan

Physics Department

Royal Holloway, University of London



Discussion on significance


The standard way to quantify the significance of a discovery

is to give the p-value of the background-only hypothesis H0:

p = Prob( data equally or more incompatible with H0 | H0 )

Requires a definition of what data values constitute a lesser

level of compatibility with H0 relative to the level found with

the observed data.

Define this to get high probability to reject H0 if a

particular signal model (or class of models) is true.

Note that actual confidence in whether a real discovery is made

depends also on other factors, e.g., plausibility of signal, degree

to which it describes the data, reliability of the model used to

find the p-value. p-value is really only first step!

Discussion on significance

Significance from p-value

Often define significance Z as the number of standard deviations

that a Gaussian variable would fluctuate in one direction

to give the same p-value.



Z = 5 corresponds to p = 2.87 × 10-7

Discussion on significance

Sensitivity (expected significance)

The significance with which one rejects the SM depends on

the particular data set obtained.

To characterize the sensitivity of a planned analysis, give the

expected (e.g., mean or median) significance assuming a

given signal model.

To determine accurately could in principle require an MC study.

Often sufficient to evaluate with representative (e.g. “Asimov”)


Discussion on significance

Significance for single counting experiment

Suppose we measure n events, expect s signal, b background.

n ~ Poisson(s+b)

Find p-value of s = 0 hypothesis.

data values with n ≥ nobs constitute lesser compatibility

Discussion on significance

Simple counting experiment with LR

Equivalently can write expectation value of n as

where m is a strength parameter (background-only is m = 0).

To test a value of m, construct likelihood ratio

where muhat is the Maximum Likelihood Estimator (MLE),

which we constrain to be positive:

Discussion on significance

p-value from LR

Also define

High values correspond to increasing incompatibility with m.

For discovery we are testing m = 0. We find

The p-value is

Discussion on significance

Significance from LR using c2 approx.

Here we will have:

and so the p-value is same as before. But for large enough n,

we can regard qm as continuous, and find

Furthermore, for large enough n, the distribution of qm approaches

a form related to the chi-square distribution for 1 d.o.f.

Complications arise from requirement that m be positive, but

end result simple. For test of m = 0 (discovery), significance is

Discussion on significance

Sensitivity for simple counting exp.

Find median significance from median n, which is approximately

s + b when this is sufficiently large.

Or, if using the approximate formula based on chi-square,

approximate median by substituting s + b for n (“Asimov” data)

For s << b, expanding logarithm and keeping terms to O(s2),

Discussion on significance

Simple counting exp. with bkg. uncertainty

Suppose b consists of several components, and that these are

not precisely known but estimated from subsidiary measurements:

n ~ Poisson,

mi ~ Poisson,

Likelihood function for full set of measurements is:

Discussion on significance

Profile likelihood ratio

To account for the nuisance parameters (systematics), test m

with the profile likelihood ratio:

Double hat: maximize

L for the given m

Single hats: maximize

L wrt m and b.

Important point is that qm = -2 ln l(m) still related to chi-square

distribution even with nuisance parameters (for sufficiently large

sample), so retain the simple formula for significance:

Discussion on significance

Examples from recent HN posts

From recent hypernews (Tetiana Hrynova, Xavier Prudent),

Consider s = 20.4, b = 2.5 ± 1.5. What is “correct” sensitivity?

First suppose b = 2.5 exactly, then:

1) Use MC to find median, assuming s = 20.4, of


2) Use formula based on chi-square approx. for likelihood ratio:

Good for s+b > dozen?

3) Use

Here OK for s << b, b > dozen?

Discussion on significance

Examples from recent HN posts (2)

To take into account the uncertainty in the background, need to

understand the origin of the 2.5 ± 1.5.

Is this e.g. an estimate based on a Poisson measurement?

Use profile likelihood for nuisance parameter b.

Or is it a Gaussian prior (truncated at zero) with mean 2.5, s = 1.5?

Use “Cousins-Highland”

Discussion on significance

Provisional conclusions

Key is to view p-value as the basic quantity of interest; Z is

equivalent, and all “magic formulae” are various approximations

for Z.

Also other considerations for discovery (and limits) beyond

p-value, e.g., level to which signal described by data, plausibility

of signal model, reliability of model for p-value, …

Also consider e.g. Bayes factors for complementary info.

StatForum should move towards firm recommendations on

what formulae to use where possible, but cannot investigate

every approximation – analysts must take some responsibility here.

Draft note (INT) attached to agenda on discovery significance;

will also have partner note on limits.

Discussion on significance

  • Login