Download
slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Gary C. Hill, Jessica Hodges, Brennan Hughey, Mike Stamatikos PowerPoint Presentation
Download Presentation
Gary C. Hill, Jessica Hodges, Brennan Hughey, Mike Stamatikos

Gary C. Hill, Jessica Hodges, Brennan Hughey, Mike Stamatikos

196 Views Download Presentation
Download Presentation

Gary C. Hill, Jessica Hodges, Brennan Hughey, Mike Stamatikos

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Examining the balance between optimising an analysis for best limit setting and best discovery potential Gary C. Hill, Jessica Hodges, Brennan Hughey, Mike Stamatikos University of Wisconsin, Madison PHYSTAT 2005, Oxford

  2. The problem? • What is an optimal analysis? • Do we go for best sensitivity (lowest upper limit) or for best discovery potential (high deviation away from background expectation)? • How do we optimise for each? • How does choosing one optimisation affect the potential of the other?

  3. Sensitivity • Sensitivity – choose cuts so that the average limit on the physics quantity is minimised • “Model Rejection Potential” • Gary C. Hill and Katherine Rawlins, Astroparticle Physics 19 393 (2003) • Standard optimisation for AMANDA and IceCube

  4. Assessing sensitivity Choose cuts – leaves the signal region, where we will look for an effect above background

  5. Limit on the flux constant A

  6. Limit on a cross section

  7. OPTIMISATION INDEPENDENT OF ASSUMED SOURCE STRENGTH!

  8. Example : sensitivity of IceCube to diffuse extraterrestrial neutrinos

  9. What about discovery potential? • Suppose one was interested in discovering something and not just setting best limits? • Would the optimisation be different? • What would one optimise?

  10. Characterising discovery* A “discovery” is a rejection of the background only hypothesis at a very high confidence level Of course, a small chance probability does not prove the truth of some alternative hypothesis - unlikely data under a background hypothesis doesn’t mean that some alternative hypothesis is true… P(n|H0) small doesn’t mean P(H1|n) is non-zero!

  11. * To give credit where it is due! • When I submitted this abstract, Louis Lyons mentioned that there was a paper from Giovanni Punzi, PHYSTAT2003, on this question • I had been at that talk and glanced at the paper a long time ago… but obviously things didn’t sink in totally… • Upon reading again, I realised that G.P. had come up with the same basic concept of a “least detectable flux” that one chooses cuts to minimise, that I will talk about and use here

  12. Characterising discovery* A “discovery” is a rejection of the background only hypothesis at a very high confidence level Of course, a small chance probability does not prove the truth of some alternative hypothesis - unlikely data under a background hypothesis doesn’t mean that some alternative hypothesis is true… P(n|H0) small doesn’t mean P(H1|n) is non-zero!

  13. Chance probabilities fordifferent sigmas John Kelley, UW Madison, with Mathematica

  14. How likely are we to see such low chance probabilities if there really is a signal?

  15. Least detectable signal

  16. Example

  17. Least detectable signal vs background Solid lines: average upper limit curves, 68, 90, 95, 99%

  18. MRF → MDF • Least detectable signal curves replace average upper limit curves • Compare minima for limits and discovery

  19. Astrophysical point source search

  20. Neutrino Beams: Heaven & Earth NEUTRINO BEAMS: HEAVEN & EARTH Black Hole Radiation Enveloping Black Hole p + g -> n + p+ ~ cosmic ray + neutrino -> p + p0 ~ cosmic ray + gamma

  21. Atmospheric Muons & Neutrinos astrophysicaln • “Atmospheric muons” from cosmic ray showers, penetrating to the detector from above • “Atmospheric neutrinos” from the same air showers, forming a diffuse background and calibration beam • Astrophysical neutrinos: interesting signal cosmic ray atm. n 10-4 Hz cosmic ray 102 Hz atm.m

  22. South Pole Dome road to work AMANDA Summer camp 1500 m Amundsen-Scott South Pole station 2000 m [not to scale]

  23. Astrophysical point source search • Know sky position of the source from other wavelength observations. • e.g. VHE gamma-ray people look where the x-ray people saw something, neutrino people look where the gamma-ray and/or x-ray people were successful

  24. Reconstruction angular resolution:point spread function differential, weighted with solid angle differential Gaussian signal uniform background Reconstruction response: Gaussian in angle away from true arrival direction

  25. integrated signal and background LDS, 5-σ signal aul 90% background

  26. Solid : best dashed : best MRF dotted : best MDF Curves of MRF, MDF MDF MRF

  27. bg = 10-4 inside 1 degree

  28. bg = 0.1 inside 1 degree

  29. bg = 1 inside 1 degree

  30. bg = 10 inside 1 degree

  31. bg = 30 inside 1 degree

  32. best MDF/MRF angular cut vs log10 background both MRF and MDF cut converge toward best b/s cut – 1.58 times resolution – historical “standard” choice of astrophysics community

  33. Comparison of the MDF/MRF performance Difference between best MRF cut and best MDF cut For larger backgrounds, don’t lose much on MRF by choosing to optimise for MDF and vice versa. At very low backgrounds, choose more carefully (discrete Poisson strikes again!)

  34. More to check? • Different confidence levels • how do optimisations vary with confidence levels? • should we use consistent levels in limit and discovery optimisation? • Different power settings • independent of power in some cases • not true generally • is there a general power rule for MDP/MRP? • Punzi method: use power corresponding to limit you would set if no discovery, i.e. 1-β = 1-α

  35. Search for neutrinos froma gamma-ray burst • Mike Stamatikos, UW Madison • Presentation at 29th International Cosmic Ray Conference, Pune, India, August 2005

  36. Conclusions • Model rejection/discovery* potential • independent of assumed strength of signals! • Both discovery and limit optimisations converge to a simple b/s form for large background expectations • For small backgrounds, look at MRF and MDF curves and decide what (mild?) tradeoffs you will accept in your analysis between limit and discovery potential * Also developed by Giovanni Punzi, PHYSTAT 2003