IceCube simulation with PPC

1. IceCube simulation with PPC Photon propagation code PPC: 2009 - now Photonics: 2000 – up to now

2. Photonics: conventional on CPU • First, run photonics to fill space with photons, tabulate the result • Create such tables for nominal light sources: cascade and uniform half-muon • Simulate photon propagation by looking up photon density in tabulated distributions • Table generation is slow • Simulation suffers from a wide range of binning artifacts • Simulation is also slow! (most time is spent loading the tables)

3. PPC simulation on GPU graphics processing unit execution threads propagation steps (between scatterings) photon absorbed new photon created (taken from the pool) threads complete their execution (no more photons) Running on an NVidia GTX 295 CUDA-capable card, ppc is configured with: 384 threads in 33 blocks (total of 12672 threads) average of ~ 1024 photons per thread (total of 1.3 . 107 photons per call)

4. Direct photon tracking with PPC photon propagation code • simulating flasher/standard candle photons • same code for muon/cascade simulation • using Henyey-Greenstein scattering function with <cos q>=0.8 • using tabulated (in 10 m depth slices) layered ice structure • employing 6-parameter ice model to extrapolate in wavelength • transparent folding of acceptance and efficiencies • precise tracking through layers of ice, no interpolation needed • much faster than photonics for E-2 nugen and unweighted CORSIKA: • 1000-1200 corsika files (4 sec each) in 24 hours • 2500-3500 E-2 nugen files in 24 hours IC-40 i.e., 10000 E-2 nugen files in ~3-4 days on 6-GPU cudatest in ~1 day on 3 cuda00X computers

5. Photon Propagation Code: PPC • There are 5 versions of the ppc: • original c++ • "fast" c++ • in Assembly • for CUDA GPU • icetray module All versions verified to produce identical results comparison with i3mcml http://icecube.wisc.edu/~dima/work/WISC/ppc/

6. ppc-gpu

7. ppc icetray module • moved from sandbox to http://code.icecube.wisc.edu/svn/projects/ppc/trunk/ • outdated code removed (old slower code that has not changed since 09/09) • new code has a wrapper: private/ppc/i3ppc.cxx, which compiles by cmake system into the libppc.so • it is necessary to compile an additional library libxppc.so by running make in private/ppc/gpu: • “make glib” compiles gpu-accelerated version (needs cuda tools) • “make clib” compiles cpu version (from the same sources!) • link to libxppc.so and libcudart.so (if gpu version) from build/lib directory • this library file must be loaded before the libppc.so wrapper library •  Should perhaps compile the CPU version by default by cmake system?

8. Configuration files ppc needs the following tables: wv.dat wavelength-tabulated DOM acceptance calculated from qe_dom2007a table of efficiency.h file as.dat overall DOM efficiency and parameters of the angular sensitivity polynomial expansion before running the program create a link to one of the provided files: as.nominal nominal (measured in the lab) as.holeice hole ice (corrected by hole ice model) rnd.txt table of random number multipliers for the multiply-with-carry random number generator tilt.par files describing the ice tilt tilt.dat delete unless using icemodel.sp2 (SPICE^2) icemodel.par file with 6 parameters of the icemodel icemodel.dat main ice properties table: depth/b_e(400)/a_dust(400)/\delta\tau (as in report icecube/200911002) before running the program create a link to one of the provided files: *All models* to be used with as.holeice unless otherwise specified icemodel.aha AHA model icemodel.sp1 SPICE model icemodel.sp2 SPICE^2 model icemodel.sp2+ SPICE^2+ model icemodel.sp2+n SPICE^2+ model with nominal ice icemodel.sp2x SPICE^2x model **ATTENTION** You must delete files tilt.par and tilt.dat if using AHA or SPICE, but not SPICE^2 Only SPICE^2 and higher have been fitted to use the tilted ice description. Can specify alternative directory containing these tables with the PPCTABLESDIR env. variable:  this could be a part of simprod or a separate project (for consistency), or part of ppc

9. Other configuration parameters A few configuration parameters may be set as “#define”s in private/ppc/gpu/ini.cxx file: #define OFLA // omit the flasher DOM #define ROMB // use rhomb cells aligned with the array #define ASENS // enable angular sensitivity #define TILT // enable tilted ice layers #define MKOW // use Marek Kowalski's photon yield parametrization #define MLTP 30 // number of multiprocessors #define WNUM 33 // optimized for 30 multiprocessors #define OVER 10 // size of photon bunches along the muon track #define NTHR 384 // NTHR*NBLK should not exceed the count of different random number multipliers #define NPHO 1024 // maximum number of photons propagated by one thread #define HNUM 1048576 // size of the output hit buffer, must hold hits from up to NPHO*NTHR*NBLK photons #define MAXLYS 180 // maximum number of ice layers #define MAXGEO 5200 // maximum number of OMs #define OVR 5 // over-R: DOM radius "oversize" scaling factor

10. Outlook • ppc has been used to simulate lots of IC40 data with V02-03-00 of simulation •  simulates 1 10000-file nugen E-2 set (sufficient for an entire analysis) in 1 day on 3 cuda001/2/3 computers (18 GPUs) • need to: • verify that it works for V02-04-00 of simulation • add code to treat high-efficient DOMs correctly • verify that it works for IC59 • improve flasher simulation (interface with photoflash) • figure out the best way to compile