PHYSTAT05 Highlights: Statistical Problems in Particle Physics, Astrophysics and Cosmology

Download Presentation

PHYSTAT05 Highlights: Statistical Problems in Particle Physics, Astrophysics and Cosmology

Loading in 2 Seconds...

- 105 Views
- Uploaded on
- Presentation posted in: General

PHYSTAT05 Highlights: Statistical Problems in Particle Physics, Astrophysics and Cosmology

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

1

PHYSTAT05 Highlights:Statistical Problems in Particle Physics, Astrophysics and Cosmology

Phystat05 Highlights

University College London

03/11/2006

Müge Karagöz Ünel

Oxford University

MKU

2

- Conference Information and History
- Introduction to statistics
- Selection of hot topics
- Available tools
- Astrophysics and cosmology
- Conclusions

Phystat05 Highlights

MKU

3

Phystat05 Highlights

MKU

4

Phystat05 Highlights

Poster

MKU

5

Phystat05 Highlights

MKU

6

7 Invited talks by Statisticians

9 Invited talks by Physicists

38 Contributed talks

8 Posters

Panel Discussion

3 Conference Summaries

90 participants

Phystat05 Highlights

MKU

7

David CoxKeynote Address: Bayesian, Frequentists & Physicists

Steffen LauritzenGoodness of Fit

Jerry FriedmanMachine Learning

Susan Holmes Visualisation

Peter CliffordTime Series

Mike TitteringtonDeconvolution

Nancy ReidConference Summary (Statistics)

Phystat05 Highlights

MKU

8

Bob CousinsNuisance Parameters for Limits

Kyle CranmerLHC discovery

Alex Szalay Astrophysics + Terabytes

Jean-Luc Starck Multiscale geometry

Jim LinnemannStatistical Software for Particle Physics

Bob NicholStatistical Software for Astrophysics

Stephen JohnsonHistorical Transits of Venus

Andrew JaffeConference Summary (Astrophysics)

Gary FeldmanConference Summary (Particles)

Phystat05 Highlights

MKU

9

Bayes/Frequentist5 talks

Goodness of Fit5

Likelihood/parameter estimation6

Nuisance parameters/limits/discovery10

Machine learning7

Software8

Visualisation1

Astrophysics5

Time series1

Deconvolution3

Phystat05 Highlights

MKU

10

Phystat05 Highlights

MKU

11

An experiment goes through following stages:

- Prepare conditions for taking data for a particle X ( if theory driven)
- Record events that might be X and reconstruct the measurables
- Select events that could have X by applying criteria (cuts)
- Generate histograms of variables and ask the questions:
Is there any evidence for new things or is the null hypothesis unrefuted? If there is evidence, what are the estimates for parameters of X?(Confrontation of theory with experiment or v.v.)

- The answers can come via your favorite statistical technique (depends on how you ask the question)

Phystat05 Highlights

MKU

12

- Homo apriorius establishes probability of an hypothesis, no matter what data tell.
- Homo pragamiticus establishes that it is interested by the data only.
- Homo frequentistus measures probability of the data given the hypothesis.
- Homo sapiens measures probability of the data and of the hypothesis.
- Homo bayesianis measures probability of the hypothesis, given the data.

Phystat05 Highlights

MKU

13

We need to make a statement about Parameters, given Data

Bayes 1763 Frequentism 1937

Both analyse data (x) statement about parameters ( )

Both use Prob (x; ), e.g. Prob ( ) = 90%

but very different interpretation

Phystat05 Highlights

Bayesian :Probability (parameter, given data)

Frequentist :Probability (data, given parameter)

“Bayesians address the question everyone is interested in, by using assumptions no-one believes”

“Frequentists use impeccable logic to deal with an issue of no interest to anyone”

MKU

14

LauritzenInvited talk - GoF

Yabsley GoF and sparse multi-D data

IanniGoF and sparse multi-D data

RajaGoF and L

Gagunashvili2and weighting

PiaSoftware Toolkit for Data Analysis

Block Rejecting outliers

Bruckman Alignment

BlobelTracking

Phystat05 Highlights

MKU

15

- We would like to: know if a given distribution is of a specified type, test the validation of a postulated model,..
- A few GoF tests are widely used in practice:
- 2 test: most widely used application is 1 or 2D fits to data
- G2 (the likelihood ratio statistics) test: the general version of 2 test (Lauritzen’s personal choice)
- Kolmogorov-Smirnov test: a robust but prone to mislead test, can be used to confirm, say, two distributions (histograms) are the same by calculating the p-value for the difference hypothesis.
- Other new methods, like Aslan&Zech’s energy test, exist…

Phystat05 Highlights

MKU

track

Intrinsic measurement error + MCS

hit

residual

Key relation!

16

Direct Least-Squares solution to the Silicon Tracker alignment problem

The method consists of minimizing the giant 2resulting from a simultaneous fit of all particle trajectories and alignment parameters:

Let us consequently use the linear expansion (we assume all second order derivatives are negligible). The track fit is solved by:

Phystat05 Highlights

while the alignment parameters are given by:

Systems large: inherent

Computational challenges

MKU

Equivalent to Millepede approach from V. Blobel

17

Cousins Limits and Nuisance Params

Reid Respondent

Punzi Frequentist multi-dimensional ordering rule

Tegenfeldt Feldman-Cousins + Cousins-Highland

Rolke Limits

Heinrich Bayes + limits

Bityukov Poisson situations

Hill Limits v Discovery (see Punzi @ PHYSTAT2003)

Cranmer LHC discovery and nuisance parameters

Phystat05 Highlights

MKU

18

Note:Systematic errors (HEP) <-> nuisance params (statistician)

An example:

we need to know these, probably from other measurements (and/or theory)

Uncertainties error in

Phystat05 Highlights

Physics parameter

Observed

for statistical errors

Some are arguably statistical errors

MKU

19

- Nuisance parameters are parameters with unknown true values. They may be:
- statistical, such as number of background events in a sideband used for estimating the background under a peak.
- systematic, such as the shape of the background under the peak, or the error caused by the uncertainty of the hadronic fragmentation model in the Monte Carlo.
- Most experiments have a large number of systematic uncertainties.
- If the experimenter is blind to these uncertainties, they become a bigger nuisance!

Phystat05 Highlights

MKU

20

- LHC will collide 40 million times/sec and collect petabytes of data. pp collisions at 14 TeV will generate events much more complicated than LEP, TeVatron.
- Kyle Cranmer has pointed out that systematic issues will be even more important at the LHC.
- If the statistical error is O(1) and systematic error is O(0.1), it does not much matter how you treat it.
- However, at the LHC, we may have processes with 100 background events and 10% systematic errors, this is not negligible.
- Even more critical, we want 5s for a discovery level.

Phystat05 Highlights

MKU

21

- LHC searches: 500 searches each of which has 100 resolution elements (mass, angle bins, ...) = 5 x 104 chances to find something.
- One experiment: False positive rate at 5 s(5 x 104) (3 x 10-7) = 0.015. OK.
- Two experiments:
- Assume allowable false positive rate: 10.
- 2 (5 x 104) (1 x 10-4) = 10 3.7 s required.
- Required other experiment verification, assume rate 0.01: (1 x 10-3)(10) = 0.01 3.1 s required.

- Caveats: Is the significance real? Are there common systematic errors?

Phystat05 Highlights

MKU

22

- Various techniques discussed during conference. Most concerns were summarized by Feldman.
- Bayesian: good method but Heinrich showed that flat priors in multi-D may lead to undesirable results (undercoverage).
- Frequentist-Bayesian hybrids: Bayesian for priors and frequentist to extract range. Cranmer considered this for LHC (which was also used at Higgs searches).
- Profile likelihood: shown by Punzi to have issues when distribution is Poisson-like.
- Full Neyman reconstruction: Cranmer and Punzi attempted this, but is not feasible for large number of nuisance parameters.

- Banff workhsop of this summer was found useful in comparing various methods. The real suggestions for LHC will likely come from 2007 workshop on LHC issues.

Phystat05 Highlights

MKU

23

- The problem: Given a measurement of an event X find F(X) which returns 1 if the event is signal (s) and 0 if the event is background (b) to optimize a figure of merit, say, s/√b for discovery and s/ √(s+b) for established signal.
- Theoretical solution: Use MC to calculate the likelihood ratio Ls(X)/Lb(X) and derive F(X) from it. Unfortunately, this does not work as in a high-dimension space, even the largest data set is sparse. (Feldman)
- In recent years, physicists have turned to machine learning: give the computer samples of s and b events and let the computer figure out what F(X) is.

Phystat05 Highlights

MKU

24

FriedmanMachine learning

ProsperRespondent

Narsky Bagging

RoeBoosting (Miniboone)

Gray Bayes optimal classification

Bhat Bayesian networks

Sarda Signal enhancement

Phystat05 Highlights

MKU

25

Various methods exist to classify, train and test events.

- Artificial neural networks (ANN): currently the most widely used (examples from Prosper, …)
- Decision trees: differentiating variable is used to separate sample into branches until a leaf with a preset number of signal and background events are found.
- Trees with rules: combining a series of trees to increase single decision tree power (Friedman)
- Bagging (Bootstrap AGGregatING) trees: build a collection of trees by selecting a sample of the training data (Narsky)
- Boosted trees: a robust method that gives misclassified events in one tree a higher weight in the generation of a new tree
Comparisons of significance were performed, but not all of were controlled experiments, so conclusions may be deceptive until further tests..

Phystat05 Highlights

MKU

Boosting the tree

Decision tree

26

- An nice example from MiniBoone
- Create M many trees and take the final score for signal and background as weighted sum of individual trees

Phystat05 Highlights

MKU

27

Giovanni Punzi @ PHYSTAT2003

“Comments on L fits with variable resolution”

Separate two close signals (A and B) , when resolution σvaries event by event, and is different for 2 signals

e.g. M, Different numbers of tracks different σM

Avoiding Punzi bias

- Include p(σ|A) and p(σ|B) in fitOR
- Fit each range of σi separately, and add (NA)i (NA)total, and similarly for B
Beware of event-by-event variables and construct likelihoods accordingly

(Talk by Catastini)

Phystat05 Highlights

MKU

28

Potential problem: Experimenters’ bias

Original suggestion? Luis Alvarez

Methods of blinding:

- Keep signal region box closed
- Add random numbers to data
- Keep Monte Carlo parameters blind
- Use part of data to define procedure
A number of analyses in experiments doing blind searches

Don’t modify result after unblinding, in general..

Question: Will LHC experiments choose to be blind? In which analysis?

Phystat05 Highlights

MKU

29

Phystat05 Highlights

MKU

Cosmologists

“Astronomers”

Particle Physicists

Bayesians

Frequentists

30

‘“There is only one universe” and some experiments can never be rerun’ – A. Jaffe (concluding talk)

Astro+cosmo tend to be more Bayesian, by nature.

Phystat05 Highlights

- Virtual Observatories: all astro data available from desktop
- Data volume growth doubling every year, most data are on the web (Szalay)
- Bad: computing & storage issues
- Good (?): Systematic errors more significant statistical errors

- Nichol discussed using grid techniques.

MKU

31

- Flat priors have been used commonly, but are dangerous (Cox, Le Diberder, Cousins): would be the best quantity to use or is it h2 ?
- Issues with non-gaussian distribution of noise taken into account in the spectrum: a few methods discussed by Starck, Digel, ..
- Blind analyses are rare (not so good at a priori modeling!)
- Lots of good software in astrophysics and repositories more advanced than PP.
- Jaffe’s talk has a a nice usage of CMB as a case study for statistical methods in astrophysics, starting from 1st principles of Bayesian.

Phystat05 Highlights

MKU

32

Phystat05 Highlights

MKU

33

LinnemannSoftware for Particles

NicholSoftware for Astro (and Grid)

Le DiberdersPlot

Paterno R

Kreschuk ROOT

Verkerke RooFit

Pia Goodness of Fit

Buckley CEDAR

Narsky StatPatternRecognition

Phystat05 Highlights

MKU

34

- A number of good software has become more and more available (good news for LHC!)
- PP and astro use somehow different softwares (IDL, IRAF by astro, for ex.)
- 2004 Phystat workshop at MSU on statistical software (mainly on R & ROOT) by Linnemann
- Statatisticians have a repository of standard source codes (StatLib): http://lib.stat.cmu.edu/
- One good output of the conference was a Recommendation of Statistical Software Repository at FNAL
- Linnemann has a web page of collections: http://www.pa.msu.edu/people/linnemann/stat_resources.html

Phystat05 Highlights

MKU

35

- Documentation about statistics and a repository: http://www-cdf.fnal.gov/physics/statistics/statistics_home.html

Phystat05 Highlights

MKU

36

Phystat05 Highlights

MKU

37

Phystat05 Highlights

MKU

38

- Very useful physicists/statisticians interaction
- e.g. Confidence intervals with nuisance parameters,
- Multivariate techniques, etc..
- Lots of things learnt from
- ourselves (by having to present own stuff!)
- each other (various different approaches..)
- statisticians (update on techniques..)

- A step towards common tools/Software repositories: http://www.phystat.org(Linnemann)
- Programme, transparencies, papers, etc:http://www.physics.ox.ac.uk/phystat05(with useful links such as recommended readings)
- Proceedings published by Imperial College Press (Spring ’06)

Phystat05 Highlights

MKU

39

- A few workshops/schools took place since October, 2005
- e.g. Manchester (Nov 2005), SAMSI Duke (April 2006), Banff (July 2006), Spanish Summer School (July 2006)
- No PHYSTAT Conference in summer 2007
- ATLAS Workshop on Statistical Methods, 18-19 Jan 2007
- PHYSTAT Workshop at CERN, 27-29 June 2007 on
- “Statistical issues for LHC Physics analyses”.
- (Both workshops will likely aim at discovery significance. Please attend!)
- Suggestions/enquiries to: l.lyons@physics.ox.ac.uk

Phystat05 Highlights

- LHC will take data soon. We do not wish to say
- rather say

“The experiment was inconclusive, so we had to use statistics”

(inside cover of “the Good Book” by L. Lyons)

We used statistics, and so we are sure that we’ve discovered X

(well… with some confidence level!)

MKU

40

- Tried to give you a collage of PHYSTAT05 topics.
- My deepest thanks to Louis for giving me the chance & introducing me to the PHYSTAT experience!
- Apologies to those talks I have not been able to cover…
- Thank you for the invitation!

Phystat05 Highlights

MKU

41

- Bayes
- Frequentist
- Cousins-Highland
- Higgs Saga at CERN

Phystat05 Highlights

MKU

42

Bayesian

Bayes’ Theorem

Phystat05 Highlights

posterior

likelihood

prior

Problems: P(param) True or False

“Degree of belief”

Prior What functional form?

Flat? Which variable?

MKU

43

Neyman Construction

µ

x

x0

µ = Theoretical parameter

x = ObservationNO PRIOR

Phystat05 Highlights

MKU

44

at 90% confidence

Frequentist

Phystat05 Highlights

Bayesian

MKU

45

Method: Mixed Frequentist - Bayesian

Full frequentist method hard to apply in several dimensions

Bayesian for nuisance parameters and

Frequentist to extract range

Philosophical/aesthetic problems?

Highland and Cousins

NIM A320 (1992) 331

Phystat05 Highlights

MKU

46

P (Data;Theory) P (Theory;Data)

Is data consistent with Standard Model?

or with Standard Model + Higgs?

Phystat05 Highlights

End of Sept 2000: Data not very consistent with S.M.

Prob (Data ; S.M.) < 1% valid frequentist statement

Turned by the press into: Prob (S.M. ; Data) < 1%

and therefore Prob (Higgs ; Data) > 99%

i.e. “It is almost certain that the Higgs has been seen”

MKU