upper limits and discovery in search for exotic physics n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Upper Limits and Discovery in Search for Exotic Physics PowerPoint Presentation
Download Presentation
Upper Limits and Discovery in Search for Exotic Physics

Loading in 2 Seconds...

play fullscreen
1 / 30

Upper Limits and Discovery in Search for Exotic Physics - PowerPoint PPT Presentation


  • 122 Views
  • Uploaded on

Upper Limits and Discovery in Search for Exotic Physics. Jan Conrad Royal Institute of Technology (KTH) Stockholm. Outline. Discovery Confidence Intervals The problem of nuisance parameters (“systematic uncertainties”) Averaging Profiling Analysis optimization Summary.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Upper Limits and Discovery in Search for Exotic Physics' - kalb


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
upper limits and discovery in search for exotic physics
Upper Limits and Discovery in Search for Exotic Physics

Jan Conrad

Royal Institute of Technology (KTH)

Stockholm

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20061

outline
Outline
  • Discovery
  • Confidence Intervals
  • The problem of nuisance parameters (“systematic uncertainties”)
    • Averaging
    • Profiling
  • Analysis optimization
  • Summary

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20062

general approach to claiming discovery hypothesis testing
General approach to claiming discovery (hypothesis testing)
  • Assume an alleged physics process characterized by a signal parameter s (flux of WIMPS, Micro Blackholes .... etc.)
  • One can claim discovery of this process if the observed data is very unlikely to come from the null hypothesis , H0, being defined as non-existence of this process (s=0). ”Very unlikely” is hereby quantified as the ”signifcance” probability αsign, taken to be a small number (often 5 σ ~ 10-7).
  • Mathematically this is done by comparing the p-value with αsign and reject H0 if p –value < αsign

Actually observed value of the test statistics

test statistics, T, could be for example χ2

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20063

p values and the neyman pearson lemma
P-values and the Neyman Pearson lemma
  • Uniformly most powerful test statistic is the likelihood ratio :
  • For p-values, we need to know the null-distribution of T.

Therefore it comes handy that asymptotically:

  • Often it is simply assumed that the null-distribution is χ2 but be careful !

see e.g. J.C. , presented at NuFACT06, Irvine, USA, Aug. 2006

L. Demortier, presented at BIRS, Banff, Canada, July 2006

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20064

type i type ii error and power
Type I, type II error and power
  • Type I error:Reject H0, though it is true.

Prob(Type I error) = α

  • Type II error:Accept H0, though it is false
    • Power:1 - β = 1 – Prob(Type II error)

In words: given H1, what is the probability that we will reject H0 at given

significance α ? In other words: what is the probability that we detect H1 ?

  • In designing a test, you want correct Type I error rate (this controls the number of false detections) and as large power as possible .

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20065

why 5 s
Why 5 s?
  • … traditional: we have seen 3 s significances disappear (….we also have seen 5 s signficances disappear on the other hand ….)
  • Principal reasoning (here done for the LHC):
    • LHC searches: 500 searches each of which has 100 resolution elements (mass, angle bins, etc.)  5 x 104 chances to find something.
    • One experiment: False positive rate at 5 s(5 x 104) (3 x 10-7) = 0.015. OK !
    • Two experiments:
      • Assume we want to produce < 100 unneccessary theory papers
      •  allowable false positive rate: 10.
      •  2 (5 x 104) (1 x 10-4) = 10  3.7 s required.
      • Required other experiment verification: (1 x 10-3)(10) = 0.01  3.1 s required.

It seems that the same reasoning would lead to smaller required signficance probabilities for EP searches in NT.

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20066

confidence intervals ci
Confidence Intervals (CI)
  • Instead of doing a hypothesis test, we might want to do a interval estimate on the parameter s with confidence level 100(1 – α) % (e.g. 90 %):
  • Bayesian:
  • Frequentist:
    • Invert by e.g. Neyman construction of confidence intervals (no time to explain)
      • - special case 1: n 2=  upper limit
      • - special case 2: two sided/one sided limits depending
      • on observation  Feldman & Cousins
  • Confidence intervals are often used for hypothesis testing.

G. Feldman & R. Cousins, Phys. RevD57:3873-3889

See e.g. J.C. presented at NuFACT06, Irvine, USA, Aug. 2006

K. S. Cranmer, PhyStat 2005, Oxford, Sept. 2005

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20067

nuisance parameters 1
Nuisance parameters1)
  • Nuisance parameters are parameters which enter the data model, but which are not of prime interest (expected background, estimated signal/background efficiencies etc. pp., often called systematic uncertainties)
  • You don’t want to give CIs (or p-values) dependent on nuisance parameters  need a way to get rid of them

1) Applies to both confidence intervals and nuisance parameters

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20068

how to get rid of the nuisance parameters
How to ”get rid” of the nuisance parameters ?
  • There is a wealth of approaches to dealing with nuisance parameters. Two are particularly common:
    • Averaging (either the likelihood or the PDF):
    • Profiling (either the likelihood or the PDF):
    • ... less common, but correct per construction: fully frequentist, see e.g:

Bayesian

G. Punzi, PHYSTAT 2005, Oxford, Sept. 2005

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 20069

nt searches for ep why things are bad and good
NT searches for EP: why things are bad ..... and good.
  • Bad
    • Low statistics makes the use of asymptotic methods doubtful
    • systematic uncertainties are large.
  • Good:
    • Many NT analyses are single channel searches with relatively few nuisance parameters
      •  rigorous methods are computationally feasible (even fully frequentist)

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200610

coverage

1 -α

1

over-covering

0.9

under-covering

s

Coverage
  • A method is said to have coverage (1-α) if, in infinitely many repeated experiments the resulting CIs include (cover) the true value in a fraction (1-α) of all cases (irrespective of what the true value is).
  • Coverage is a necessary and sufficient condition for a valid CI calculation method

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200611

averaging hybrid bayesian confidence intervals
Averaging: hybrid Bayesian confidence intervals
  • Example PDF:
  • Perform Neyman-Construction with this new PDF (we will assume Feldman & Cousins in the remainder of this talk)
  • Treats nuisance parameters Bayesian, but performs a frequentist construction.

Integral is performed in true variables  Bayesian

J.C, O. Botner, A. Hallgren, C. de los Heros Phys. RevD67:012002,2003

R. Cousins & V. Highland Nucl. Inst. Meth. A320:331-335,1992

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200612

coverage of hybrid method
Coverage of hybrid method.

Use Log-normal if large uncertainties !!!!!

(1- α)MC

true s

true s

F.Tegenfeldt & J.C. Nucl. Instr. Meth.A539:407-413, 2005

J.C & F. Tegenfeldt , PhyStat 05, Oxford, Sept. 2005, physics/0511055

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200613

commercial break pole
Commercial break: pole++
  • Bayesian treatment in FC ordering Neyman construction
    • treats P(n|εs +b)
  • Consists of C++ classes:
    • Pole calculate limits
    • Coverage coverage studies
    • Combine combine experiments
  • Nuisance parameters
    • supports flat, log-normal and Gaussian uncertainties in efficiency and background
    • Correlations (multi-variate distributions and uncorrelated case)
  • Code and documentation available from:
    • http://cern.ch/tegen/statistics.html

J.C & F. Tegenfeldt , Proceedings PhyStat 05, physics/0511055

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200614

example hybrid bayesian in nts
Example: hybrid Bayesian in NTs
  • From Daan Huberts talk (this conference):

with systematicswithout systematics

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200615

profiling profile likelihood confidence intervals
Profiling: Profile Likelihood confidence intervals

meas n, meas. b

MLE of b given s

MLE of b and s given observations

2.706

To extract limits:

Lower limit

Upper Limit

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200616

from minuit manual
From MINUIT manual
  • See F. James, MINUIT Reference Manual, CERN Library Long Write-up D506, p.5:

“The MINOS error for a given parameter is defined as the change in the value of the parameter that causes the F’ to increase by the amount UP, where F’ is the minimum w.r.t to all other free parameters”.

Confidence Interval

ΔΧ2 = 2.71 (90%),

ΔΧ2 = 1.07 (70 %)

Profile Likelihood

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200617

coverage of profile likelihood

Background: Poisson (unc ~ 20 % -- 40 %) , Efficiency: binomial (unc ~ 12%)

Rolke et al

Minuit

Coverage of profile likelihood

Available as TRolke in ROOT !

Should be able to treat common NT cases

(1- α)MC

W. Rolke, A. Lopez, J.C. Nucl. Inst.Meth A 551 (2005) 493-503

true s

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200618

profile likelihood goes lhc
Profile likelihood goes LHC.
  • Basic idea: calculate 5 σ confidence interval and claim discovery if s = 0 is not included.
  • Straw-man model:
  • Typical: b = 100, т = 1 ( 10 % sys. Uncertainty on b)

Size of side band region

- 35 events!!

- 17 events!!

K. S. Cranmer, PHYSTAT 2005, Oxford, Sept. 2005

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200619

analysis optimisation
Analysis optimisation
  • Consider some cut-value t. Analysis is optimised defining a figure of merit (FOM). Very common:
  • Alternatively, optimize for most stringent upper limit. The corresponding figure of merit is the model rejection factor, MRF:

Mean upper limit (only bg)

G. Hill & K. Rawlins, Astropart. Phys. 19:393-402,2003

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200620

in case of systematics
In case of systematics ?
  • Simplest generalizations one could think of:
  • In general, I do not think it makes a difference unless:

NO !

Yes !

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200621

optimisation for discovery and upper limit at the same time
Optimisation for discovery and upper limit at the same time ?
  • Fix significance (e.g αsign = 5 σ) and confidence level (e.g. 1-αCL = 99 %). Then define sensitivity region in s by :
  • The FOM can be defined to optimize this quantity (e.g simple counting experiment):

Signal efficiency

Number of σ (here assumed αsign = 1 – αCL)

G. Punzi, PHYSTAT 2003, SLAC, Aug. 2003

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200622

conclusions final remarks
Conclusions/Final Remarks
  • Two methods to calculate CI and claim discovery in presence of ”systematic” uncertainties have been discussed.
  • The methods presented here are certainly suitable for searches for Exotic Physics with Neutrino Telescopes and code exists which works ”out of the box”
    • Remark: the ”simplicity” of the problem (single channel, small number of nuisance parameters) make even rigorous methods applicable
    • Remark 2: the LHC example shows that for large signficances (discovery) hybrid Bayesian might be problematic.
  • I discussed briefly the issue of sensitivity and analysis optimisation.

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200623

slide24
Backup Slides

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200624

b 0 s
B0s µ+µ-

J.C & F. Tegenfeldt , Proceedings PhyStat 05, physics/0511055

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200625

slide26

Neyman construction

Exp 3

Exp 2

Exp 1

One additional degree of freedom: ORDER in which you inlcude the n into the belt

J. Neyman, Phil. Trans. Roy. Soc. London A, 333, (1937)

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200626

projection method with appropriate ordering

s

Projection method with appropriate ordering.

Ordering function: (Punzi, PhyStat05)

Poisson signal, Gauss eff. Unc (10 %)

Can be any ordering in prime observable sub-space, in this case

Likelihood ratio (Feldman & Cousins)

~ FC Profile

Average coverage

Max/Min coverage

s

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200627

fc ordering coverage
FC ordering: coverage

(1- α)MC

Calculated by Pseudo-experiments

Nominal coverage

true s

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200628

some methods for p value calculation
Some methods for p-value calculation
  • Conditioning
  • Prior-predictive
  • Posterior-predictive
  • Plug-In
  • Likelihood Ratio
  • Confidence Interval
  • Generalized frequentist

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200629

some methods for confidence interval calculation the banff list
Some methods for confidence interval calculation (the Banff list)
  • Bayesian
  • Feldman & Cousins with Bayesian treatment of nuisance parameters (Hybrid Bayesian)
  • Profile Likelihood
  • Modified Likelihood
  • Feldman & Cousins with Profile Likelihood
  • Fully frequentist
  • Empirical Bayes

Jan Conrad (KTH, Sthlm) EPNT 06 20 September 200630