Report from the Statistics Forum

1. Report from the Statistics Forum ATLAS Analysis Strategies Workshop CERN, 6 September, 2007 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan Analysis Strategies Workshop -- Statistics Forum Report

2. The ATLAS Statistics Forum Initiated at Statistics Workshop in January 07 Statistics Forum started in March (Karl Jakobs) Since then, meetings ~monthly (plus ~3 joint meetings with CMS) Analysis Strategies Workshop -- Statistics Forum Report

3. Current Membership Conveners: Eilam Gross, Glen Cowan “Statistics Experts”: GDC, Kyle Cranmer, Tom Junk, Bill Murray, Alex Read Tool developers: Bill Quayle, Krzysztof Ciba Representatives from ATLAS groups: Higgs: Eilam Gross & Guillaume Unal SUSY: Dan Tovey SM: Maarten Boonekamp Exotics: Samir Ferrag B-Physics: Eduard Burelo Top: Yoram Rozen Heavy Ions, ... Ex officio: Ian Hinchliffe & Karl Jakobs Analysis Strategies Workshop -- Statistics Forum Report

4. Mandate Define ATLAS Statistics Standards in contact with Physics Coordination and ATLAS groups. Develop, validate and approve standard statistical tools in close contact with the software group. Represent ATLAS in discussions on statistical issues with CMS. Discuss and make recommendations on statistical issues such as blind analyses, model independent searches, etc. Analysis Strategies Workshop -- Statistics Forum Report

5. Web page https://twiki.cern.ch/twiki/bin/view/Atlas/StatisticsTools Analysis Strategies Workshop -- Statistics Forum Report

6. Hypernews https://hypernews.cern.ch/HyperNews/Atlas/get/physics-statistics.html Analysis Strategies Workshop -- Statistics Forum Report

7. cf. BaBar Statistics Group Hypernews Detailed discussions Heavy traffic Analysis Strategies Workshop -- Statistics Forum Report

8. Some ongoing activities ATLAS Statistics Book Higgs combination (different channels & ATLAS + CMS) Software development (RooStat) Sequential analysis Analysis Strategies Workshop -- Statistics Forum Report

9. Twiki, i.e., a Living Document (thanks, Krzysztof) Core authors assigned, detailed outline discussed + worked examples, particularly related to systematics + FAQ, recommendations Goal: first draft by October (need to work hard...) Analysis Strategies Workshop -- Statistics Forum Report

10. Higgs Combination Higgs suggested as a first major exercise for the group, but we would like to undertake similar efforts with other Physics Groups. Many people involved; see next talk by Eilam. In particular, frequentist combination based on profile likelihood very advanced (Kyle, Eilam, Ofer, ...) But a brief comment on the Bayesian methods: Bayesian limits already widely used e.g. in CDF (→Tom J.). We are also investigating use of Bayes factors as a means of quantifying significance of a discovery: Higgs posterior odds prior odds Bayes factor B10 no Higgs Analysis Strategies Workshop -- Statistics Forum Report

11. Higgs combination based on event counts N independent channels, count ni events in search regions: Constrain expected background bi with sideband measurements: Expected number of signal events: (m is global parameter, m = 1 for SM). Consider a fixed Higgs mass and assume SM branching ratios Bi. Suggested method: constrain m with limit mup; consider mH excluded if upper limit mup < 1.0. For discovery, compute Bayes factor for H0 : m = 0 vs. H1 : m = 1 Analysis Strategies Workshop -- Statistics Forum Report

12. Bayes factor ( = ratio of integrated likelihoods) B10 at m = 1 gives change in odds of SM Higgs vs. no Higgs in light of data, i.e., measures of weight of evidence provided by data in support of one hypothesis over another. B10 Here B10 and profile likelihood ratio close; comparisons ongoing. Analysis Strategies Workshop -- Statistics Forum Report

13. Statistical software Bill Quayle, Kyle Cranmer (from Statistics Forum) + developers of TMVA (Andreas Hoecker et al.), RooFit (Wouter Verkerke et al.), MadTools/StatTools (Wisconsin group), ... (see e.g. Bill Quayle’s 15 March 07 talk) Some issues: ATLAS specific (private) or HEP-wide (public)? Use FrameworkXXX or, integrate ComponentXXX into an existing framework? Distribution (phystat.org, bundle with ROOT?) Analysis Strategies Workshop -- Statistics Forum Report

14. RooFit/RooStats Wouter Verkerke, Kyle Cranmer Developments reported in ATLAS/CMS Forum, i.e., HEP-wide, not ATLAS-specific software. E.g. Combination Toy Model (W. Verkerke -- 17 July 2007 ATLAS/CMS meeting) Combined Profile likelihood ‘Atlas’ → talk by Wouter ‘CMS’ Analysis Strategies Workshop -- Statistics Forum Report

15. (Renaud Bruneliere, 7 June and 3 July 07) Sequential analysis Looking at data as they are recorded and compute a new p-value or likelihood ratio test every 10 pb-1. What is probability that p-value (or LR test) < 0.01 versus number of data looks ? p-Value result versus L when H0 is true P(at least 1 p-Value < 0.01) versus L when H0 is true b = 200 fb Likelihood ratio test p-value • looking 1 time at data : p(p-value < 0.01|H0) = 0.01 • looking sequentially 100 times at data : p(at least 1 p-value < 0.01|H0) ~ 0.08 ! →‘Look-elsewhere effect in time’. Need to understand how to best quantify / deal with this!

16. Conclusions, thoughts Statistics Forum working since March, several projects ongoing: Statistics Book Higgs combination Software tools Sequential analysis Want to increase interaction with Physics Groups, If your group has a statistics issue/problem, is developing a statistical tool, etc. come tell us. (Also we will come to you...) Analysis Strategies Workshop -- Statistics Forum Report