Phystat05 highlights statistical problems in particle physics astrophysics and cosmology
1 / 46

PHYSTAT05 Highlights: Statistical Problems in Particle Physics, Astrophysics and Cosmology - PowerPoint PPT Presentation

  • Uploaded on

1. PHYSTAT05 Highlights: Statistical Problems in Particle Physics, Astrophysics and Cosmology. Phystat05 Highlights. University College London 03/11/2006. M ü ge Karag ö z Ü nel Oxford University. MKU. 2. Outline. Conference Information and History Introduction to statistics

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' PHYSTAT05 Highlights: Statistical Problems in Particle Physics, Astrophysics and Cosmology' - wendi

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Phystat05 highlights statistical problems in particle physics astrophysics and cosmology


PHYSTAT05 Highlights:Statistical Problems in Particle Physics, Astrophysics and Cosmology

Phystat05 Highlights

University College London


Müge Karagöz Ünel

Oxford University





  • Conference Information and History

  • Introduction to statistics

  • Selection of hot topics

  • Available tools

  • Astrophysics and cosmology

  • Conclusions

Phystat05 Highlights


Phystat history



Phystat05 Highlights



Phystat05 Highlights



Chronology of phystat05


Chronology of PHYSTAT05

Phystat05 Highlights


Phystat05 programme


PHYSTAT05 Programme

7 Invited talks by Statisticians

9 Invited talks by Physicists

38 Contributed talks

8 Posters

Panel Discussion

3 Conference Summaries

90 participants

Phystat05 Highlights


Invited talks by statisticians


Invited Talks by Statisticians

David Cox Keynote Address: Bayesian, Frequentists & Physicists

Steffen Lauritzen Goodness of Fit

Jerry Friedman Machine Learning

Susan Holmes Visualisation

Peter Clifford Time Series

Mike Titterington Deconvolution

Nancy Reid Conference Summary (Statistics)

Phystat05 Highlights


Invited talks by astro physicists


Invited Talks by (Astro+)Physicists

Bob Cousins Nuisance Parameters for Limits

Kyle Cranmer LHC discovery

Alex Szalay Astrophysics + Terabytes

Jean-Luc Starck Multiscale geometry

Jim Linnemann Statistical Software for Particle Physics

Bob Nichol Statistical Software for Astrophysics

Stephen Johnson Historical Transits of Venus

Andrew Jaffe Conference Summary (Astrophysics)

Gary Feldman Conference Summary (Particles)

Phystat05 Highlights


Contents of the proceedings


Contents of the Proceedings

Bayes/Frequentist 5 talks

Goodness of Fit 5

Likelihood/parameter estimation 6

Nuisance parameters/limits/discovery 10

Machine learning 7

Software 8

Visualisation 1

Astrophysics 5

Time series 1

Deconvolution 3

Phystat05 Highlights


Statistics in a p physics


Statistics in (A/P)Physics

Phystat05 Highlights


Statistics in particle physics


Statistics in (Particle) Physics

An experiment goes through following stages:

  • Prepare conditions for taking data for a particle X ( if theory driven)

  • Record events that might be X and reconstruct the measurables

  • Select events that could have X by applying criteria (cuts)

  • Generate histograms of variables and ask the questions:

    Is there any evidence for new things or is the null hypothesis unrefuted? If there is evidence, what are the estimates for parameters of X?(Confrontation of theory with experiment or v.v.)

  • The answers can come via your favorite statistical technique (depends on how you ask the question)

Phystat05 Highlights


Yet another chronology from s andreon s web page


(yet another) Chronology (from S. Andreon’s web page)

  • Homo apriorius establishes probability of an hypothesis, no matter what data tell.

  • Homo pragamiticus establishes that it is interested by the data only.

  • Homo frequentistus measures probability of the data given the hypothesis.

  • Homo sapiens measures probability of the data and of the hypothesis.

  • Homo bayesianis measures probability of the hypothesis, given the data.

Phystat05 Highlights


Bayesian vs frequentist


Bayesian vs Frequentist

We need to make a statement about Parameters, given Data

Bayes 1763 Frequentism 1937

Both analyse data (x)  statement about parameters (  )

Both use Prob (x;  ), e.g. Prob ( ) = 90%

but very different interpretation

Phystat05 Highlights

Bayesian : Probability (parameter, given data)

Frequentist : Probability (data, given parameter)

“Bayesians address the question everyone is interested in, by using assumptions no-one believes”

“Frequentists use impeccable logic to deal with an issue of no interest to anyone”


Goodness of fit


Goodness of Fit

Lauritzen Invited talk - GoF

Yabsley GoF and sparse multi-D data

Ianni GoF and sparse multi-D data

Raja GoF and L

Gagunashvili 2and weighting

Pia Software Toolkit for Data Analysis

Block Rejecting outliers

Bruckman Alignment

Blobel Tracking

Phystat05 Highlights


Goodness of fit1


Goodness of Fit

  • We would like to: know if a given distribution is of a specified type, test the validation of a postulated model,..

  • A few GoF tests are widely used in practice:

    • 2 test: most widely used application is 1 or 2D fits to data

    • G2 (the likelihood ratio statistics) test: the general version of 2 test (Lauritzen’s personal choice)

    • Kolmogorov-Smirnov test: a robust but prone to mislead test, can be used to confirm, say, two distributions (histograms) are the same by calculating the p-value for the difference hypothesis.

    • Other new methods, like Aslan&Zech’s energy test, exist…

Phystat05 Highlights


An example from atlas bruckman


Intrinsic measurement error + MCS



Key relation!


An example from ATLAS (Bruckman)

Direct Least-Squares solution to the Silicon Tracker alignment problem

The method consists of minimizing the giant 2resulting from a simultaneous fit of all particle trajectories and alignment parameters:

Let us consequently use the linear expansion (we assume all second order derivatives are negligible). The track fit is solved by:

Phystat05 Highlights

while the alignment parameters are given by:

Systems large: inherent

Computational challenges


Equivalent to Millepede approach from V. Blobel

Nuisance parameters limits discovery


Nuisance Parameters/Limits/Discovery

Cousins Limits and Nuisance Params

Reid Respondent

Punzi Frequentist multi-dimensional ordering rule

Tegenfeldt Feldman-Cousins + Cousins-Highland

Rolke Limits

Heinrich Bayes + limits

Bityukov Poisson situations

Hill Limits v Discovery (see Punzi @ PHYSTAT2003)

Cranmer LHC discovery and nuisance parameters

Phystat05 Highlights





Note:Systematic errors (HEP) <-> nuisance params (statistician)

An example:

we need to know these, probably from other measurements (and/or theory)

Uncertainties error in

Phystat05 Highlights

Physics parameter


for statistical errors

Some are arguably statistical errors


Nuisance parameters


Nuisance Parameters

  • Nuisance parameters are parameters with unknown true values. They may be:

    • statistical, such as number of background events in a sideband used for estimating the background under a peak.

    • systematic, such as the shape of the background under the peak, or the error caused by the uncertainty of the hadronic fragmentation model in the Monte Carlo.

    • Most experiments have a large number of systematic uncertainties.

    • If the experimenter is blind to these uncertainties, they become a bigger nuisance!

Phystat05 Highlights


Issues with lhc


Issues with LHC

  • LHC will collide 40 million times/sec and collect petabytes of data. pp collisions at 14 TeV will generate events much more complicated than LEP, TeVatron.

  • Kyle Cranmer has pointed out that systematic issues will be even more important at the LHC.

    • If the statistical error is O(1) and systematic error is O(0.1), it does not much matter how you treat it.

    • However, at the LHC, we may have processes with 100 background events and 10% systematic errors, this is not negligible.

    • Even more critical, we want 5s for a discovery level.

Phystat05 Highlights


Why 5 s feldman cranmer


Why 5s? (Feldman+Cranmer)

  • LHC searches: 500 searches each of which has 100 resolution elements (mass, angle bins, ...) = 5 x 104 chances to find something.

  • One experiment: False positive rate at 5 s(5 x 104) (3 x 10-7) = 0.015. OK.

  • Two experiments:

    • Assume allowable false positive rate: 10.

    • 2 (5 x 104) (1 x 10-4) = 10 3.7 s required.

    • Required other experiment verification, assume rate 0.01: (1 x 10-3)(10) = 0.01 3.1 s required.

  • Caveats: Is the significance real? Are there common systematic errors?

Phystat05 Highlights


Confidence intervals


Confidence Intervals

  • Various techniques discussed during conference. Most concerns were summarized by Feldman.

    • Bayesian: good method but Heinrich showed that flat priors in multi-D may lead to undesirable results (undercoverage).

    • Frequentist-Bayesian hybrids: Bayesian for priors and frequentist to extract range. Cranmer considered this for LHC (which was also used at Higgs searches).

    • Profile likelihood: shown by Punzi to have issues when distribution is Poisson-like.

    • Full Neyman reconstruction: Cranmer and Punzi attempted this, but is not feasible for large number of nuisance parameters.

  • Banff workhsop of this summer was found useful in comparing various methods. The real suggestions for LHC will likely come from 2007 workshop on LHC issues.

Phystat05 Highlights


Event classification


Event Classification

  • The problem: Given a measurement of an event X find F(X) which returns 1 if the event is signal (s) and 0 if the event is background (b) to optimize a figure of merit, say, s/√b for discovery and s/ √(s+b) for established signal.

  • Theoretical solution: Use MC to calculate the likelihood ratio Ls(X)/Lb(X) and derive F(X) from it. Unfortunately, this does not work as in a high-dimension space, even the largest data set is sparse. (Feldman)

  • In recent years, physicists have turned to machine learning: give the computer samples of s and b events and let the computer figure out what F(X) is.

Phystat05 Highlights


Multivariate analysis


Multivariate Analysis

Friedman Machine learning

Prosper Respondent

Narsky Bagging

Roe Boosting (Miniboone)

Gray Bayes optimal classification

Bhat Bayesian networks

Sarda Signal enhancement

Phystat05 Highlights


Multivariates and machine learning


Multivariates and Machine Learning

Various methods exist to classify, train and test events.

  • Artificial neural networks (ANN): currently the most widely used (examples from Prosper, …)

  • Decision trees: differentiating variable is used to separate sample into branches until a leaf with a preset number of signal and background events are found.

  • Trees with rules: combining a series of trees to increase single decision tree power (Friedman)

  • Bagging (Bootstrap AGGregatING) trees: build a collection of trees by selecting a sample of the training data (Narsky)

  • Boosted trees: a robust method that gives misclassified events in one tree a higher weight in the generation of a new tree

    Comparisons of significance were performed, but not all of were controlled experiments, so conclusions may be deceptive until further tests..

Phystat05 Highlights


Ex boosted decision trees roe

Boosting the tree

Decision tree


Ex: Boosted Decision Trees (Roe)

  • An nice example from MiniBoone

  • Create M many trees and take the final score for signal and background as weighted sum of individual trees

Phystat05 Highlights


Punzi effect getting l wrong


Punzi effect (getting L wrong)

Giovanni Punzi @ PHYSTAT2003

“Comments on L fits with variable resolution”

Separate two close signals (A and B) , when resolution σvaries event by event, and is different for 2 signals

e.g. M, Different numbers of tracks  different σM

Avoiding Punzi bias

  • Include p(σ|A) and p(σ|B) in fitOR

  • Fit each range of σi separately, and add (NA)i (NA)total, and similarly for B

    Beware of event-by-event variables and construct likelihoods accordingly

    (Talk by Catastini)

Phystat05 Highlights


Blind analyses


Blind Analyses

Potential problem: Experimenters’ bias

Original suggestion? Luis Alvarez

Methods of blinding:

  • Keep signal region box closed

  • Add random numbers to data

  • Keep Monte Carlo parameters blind

  • Use part of data to define procedure

    A number of analyses in experiments doing blind searches

    Don’t modify result after unblinding, in general..

    Question: Will LHC experiments choose to be blind? In which analysis?

Phystat05 Highlights


Astrophysics cosmology highlights


Astrophysics + Cosmology Highlights

Phystat05 Highlights


Astro cosmo general issues



Particle Physicists




Astro/Cosmo General Issues

‘“There is only one universe” and some experiments can never be rerun’ – A. Jaffe (concluding talk)

 Astro+cosmo tend to be more Bayesian, by nature.

Phystat05 Highlights

  • Virtual Observatories: all astro data available from desktop

  • Data volume growth doubling every year, most data are on the web (Szalay)

    • Bad: computing & storage issues

    • Good (?): Systematic errors more significant statistical errors

  • Nichol discussed using grid techniques.


Astrophysics various hot points


Astrophysics: Various Hot Points

  • Flat priors have been used commonly, but are dangerous (Cox, Le Diberder, Cousins): would  be the best quantity to use or is it h2 ?

  • Issues with non-gaussian distribution of noise taken into account in the spectrum: a few methods discussed by Starck, Digel, ..

  • Blind analyses are rare (not so good at a priori modeling!)

  • Lots of good software in astrophysics and repositories more advanced than PP.

  • Jaffe’s talk has a a nice usage of CMB as a case study for statistical methods in astrophysics, starting from 1st principles of Bayesian.

Phystat05 Highlights


Software and available tools


Software and Available Tools

Phystat05 Highlights


Talks given on software


Talks Given on Software

Linnemann Software for Particles

Nichol Software for Astro (and Grid)

Le Diberder sPlot

Paterno R

Kreschuk ROOT

Verkerke RooFit

Pia Goodness of Fit

Buckley CEDAR

Narsky StatPatternRecognition

Phystat05 Highlights


Available tools


Available Tools

  • A number of good software has become more and more available (good news for LHC!)

  • PP and astro use somehow different softwares (IDL, IRAF by astro, for ex.)

  • 2004 Phystat workshop at MSU on statistical software (mainly on R & ROOT) by Linnemann

  • Statatisticians have a repository of standard source codes (StatLib):

  • One good output of the conference was a Recommendation of Statistical Software Repository at FNAL

  • Linnemann has a web page of collections:

Phystat05 Highlights


Cdf statistics committee resources


CDF Statistics Committee resources

  • Documentation about statistics and a repository:

Phystat05 Highlights


Sample repository page


Sample Repository Page

Phystat05 Highlights


Cedar cepa



Phystat05 Highlights


Summary conclusions


Summary & Conclusions

  • Very useful physicists/statisticians interaction

  • e.g. Confidence intervals with nuisance parameters,

  • Multivariate techniques, etc..

  • Lots of things learnt from

    • ourselves (by having to present own stuff!)

    • each other (various different approaches..)

    • statisticians (update on techniques..)

  • A step towards common tools/Software repositories:

  • Programme, transparencies, papers, etc: useful links such as recommended readings)

  • Proceedings published by Imperial College Press (Spring ’06)

Phystat05 Highlights


What is next


What is Next?

  • A few workshops/schools took place since October, 2005

  • e.g. Manchester (Nov 2005), SAMSI Duke (April 2006), Banff (July 2006), Spanish Summer School (July 2006)

  • No PHYSTAT Conference in summer 2007

  • ATLAS Workshop on Statistical Methods, 18-19 Jan 2007

  • PHYSTAT Workshop at CERN, 27-29 June 2007 on

  • “Statistical issues for LHC Physics analyses”.

  • (Both workshops will likely aim at discovery significance. Please attend!)

  • Suggestions/enquiries to: [email protected]

Phystat05 Highlights

  • LHC will take data soon. We do not wish to say

  • rather say

“The experiment was inconclusive, so we had to use statistics”

(inside cover of “the Good Book” by L. Lyons)

We used statistics, and so we are sure that we’ve discovered X

(well… with some confidence level!)


Some final notes


Some Final Notes

  • Tried to give you a collage of PHYSTAT05 topics.

  • My deepest thanks to Louis for giving me the chance & introducing me to the PHYSTAT experience!

  • Apologies to those talks I have not been able to cover…

  • Thank you for the invitation!

Phystat05 Highlights





  • Bayes

  • Frequentist

  • Cousins-Highland

  • Higgs Saga at CERN

Phystat05 Highlights


Bayesian approach


Bayesian Approach


Bayes’ Theorem

Phystat05 Highlights




Problems: P(param) True or False

“Degree of belief”

Prior What functional form?

Flat? Which variable?


Frequentist approach


Frequentist Approach

Neyman Construction




µ = Theoretical parameter

x = ObservationNO PRIOR

Phystat05 Highlights


Frequentist approach1


Frequentist Approach

at 90% confidence


Phystat05 Highlights



A method


A Method

Method: Mixed Frequentist - Bayesian

Full frequentist method hard to apply in several dimensions

Bayesian for nuisance parameters and

Frequentist to extract range

Philosophical/aesthetic problems?

Highland and Cousins

NIM A320 (1992) 331

Phystat05 Highlights


Higgs saga


Higgs Saga

P (Data;Theory) P (Theory;Data)

Is data consistent with Standard Model?

or with Standard Model + Higgs?

Phystat05 Highlights

End of Sept 2000: Data not very consistent with S.M.

Prob (Data ; S.M.) < 1% valid frequentist statement

Turned by the press into: Prob (S.M. ; Data) < 1%

and therefore Prob (Higgs ; Data) > 99%

i.e. “It is almost certain that the Higgs has been seen”