sPHENIX Computing status, plans and review homework

1. sPHENIXComputing status, plans and review homework

2. Current RACF disk resources No change from 2017 sPHENIX $HOME: 3GB /sphenix/sim/sim01 100TB /sphenix/sim/sim02 10TB /sphenix/data/data01 5TB /sphenix/data/data02 150TB /sphenix/user 100TB Except home dir, nothing is backed up, so don’t ask. Home dir backup does not go too far back either More disk space coming soon, add dCache usage? It’s still a bit of a mess – the plan is to move user owned files to /sphenix/user and keep /sphenix/sim and /sphenix/data filesystems exclusive for phnxreco/anatrain. Check your gpfs usage and quota : https://network.racf.bnl.gov/Facility/GCE/GPFS/index.html sphenix filesystems are on gpfs02 and gpfs04, remember to divide by 2 for usable space because of dual copy Disclaimer – all resources courtesy of PHENIX

3. Current RACF Computing resources 13620 Cores (PHENIX) 6000 Cores (Shared Pool) 57700 Cores (all of RACF – more later) Special himem queue which can run jobs > 4GB memory only about 600 condor slots for himem Curious what Condor is doing: https://monitoring.sdcc.bnl.gov/grafana/?orgId=1 Other Services: Gittea – private BNL sphenix git repo for papers/proprietary code: https://git.racf.bnl.gov/gitea/sPHENIX PostgreSQL DB server: sphenixdbmaster Drastic change for condor coming, should make himem jobs easier (but unsure about number of those) Disclaimer – all resources courtesy of PHENIX

4. sPHENIX accounts sPHENIX has been part of the RHIC account request page for a while 3GB quota on /sphenix/u/<username> Shell is bash (not tcsh as in PHENIX) sPHENIX accounts are for collaborators which are not members of PHENIX – they do not provide access to PHENIX Resources (PHENIX disk space or PHENIX internal web pages). Collaborators from PHENIX institutions have the choice between a PHENIX account (with sphenix as secondary group id giving the same privileges as sPHENIX accounts) and an sPHENIX account (preferred for sPHENIX only work). We will not give sPHENIX accounts access to PHENIX resources. Please keep this distinction in mind when requesting accounts.

5. 3rd party software • 1.1 boost • 1.2 CGAL • 1.3 CLHEP • 1.4 Eic-smear • 1.5 eigen • 1.6 EvtGen • 1.7 fastjet • 1.8 fastjetcontrib • 1.9 Geant4 • 1.10 genfit • 1.11 gmp • 1.12 gprof2dot • 1.13 gsl • 1.14 HepMC • 1.15 JETSCAPE • 1.16 LHAPDF5/6 • 1.17 libodbc++ • 1.18 MPFR • 1.19 PHOTOS • 1.20 psqlodbc • 1.21 pythia6/pythia8 • 1.22 rave • 1.23 RooUnfold • 1.24 root • 1.25 Sartre • 1.26 unixODBC • 1.27 TAUOLA • 1.28 xerces Most 3rd party libraries and includes are copied to $OFFLINE_MAIN so builds stay consistent even if 3rd party software is changed We try to keep the amount of 3rd party software limited to reduce maintenance Given todays release cycles there are almost daily updates. Validation and Verification of updates is time intensive and we don’t have ready to use suites to evaluate and quantify changes Software versions were the latest as of transition to SL7 – upgraded if really needed Giving back: we have submitted bugfixes to G4, Genfit and Jetscape

6. Event generators Pythia6 pythiaeRHIC : pythia6 + radgen Pythia8 EventGen Hijing Jetscape (needs hepmc3 build) Jewel Sartre Lhapdf5/6 for pdfs Our preferred interface is HepMC, if your generator can write it – we can read it Ongoing work to switch over to HepMC 3

7. Theoreticians view of simulations

8. Interface Detector 1 Construct()  Geometry Construct()  Geometry Stepping Action (Hit extraction) Stepping Action (Hit extraction) Experimentalists view of simulations Fun4AllServer Geant4 Event generator (input file, single particle, pythia8) PHG4Reco Node tree Interface Detector 2 Digitisation sPHENIX Raw Data Setup Tracking,Clustering calls Jet Finding, Upsilons, Photons,… dataflow

9. Students view of simulations JETSCAPE - Pythia brick test example: Pythia8 event going through a brick of qgp measured by sPHENIX

10. Build flavors new : daily build of current github repo (default) new+insure : daily insure build of current github repo play : daily build of some test (currently latest G4 version) root6 : daily build with ROOT6 hepmc3 : fixed build using HepMC3 coverity : daily build with coverity (output into coverity DB, not really usable for general public) coverity-html : daily build with coverity (html output) https://www.phenix.bnl.gov/WWW/p/draft/phnxbld/sphenix/coverity/report/ doxygen : daily build to keep doxygen up to date https://www.phenix.bnl.gov/WWW/offline/doxygen_sPHENIX/html/index.html ana : weekly build of current github repo, archived We keep 4 of each daily build  they roll over every 4 days, if you do not refresh your environment with source /opt/sphenix/core/bin/sphenix_setup.csh –n <build type> you will encounter hard to recognize inconsistencies sometimes. Use the ana build if you do not need the latest and greatest – it will stay around “forever” Status of the builds in “tinderbox”: https://www.phenix.bnl.gov/software/sPHENIX/tinderbox/showbuilds.cgi?tree=default Disclaimer: The web pages are PHENIX internal – we need to change this

11. Cvmfs (CernVM-FS https://cernvm.cern.ch/portal/filesystem) The problem: afs use is declining in HEP, openafs’s maintenance is dicey (might need replacing by Auristor a commercial product). kAFS – the LINUX version is experimental, some utils exist from 2014 – probably not ready to use The solution: cvmfs global read only filesystem, widely used in the HEP community accessible on the OSG (Open Science Grid) to run PHENIX production on the OSG we copied the libraries from afs to cvmfs (courtesy of Fermi Lab) BNL has now a tier 0 cvmfs server, we are under /cvmfs/sphenix.sdcc.bnl.gov/ Cvmfs is easy to set up (I have it running on my desktop, try this with afs) About 3 weeks ago, we switched from afs to cvmfs for our builds (nobody noticed) /opt/sphenix is now a softlink to cvmfs Caveat: cvmfs has no equivalent for @sys to distinguish between OS platforms cvmfs is not an “interactive filesystem”, where users can write to (and read back immediately) We still build daily the afs version but will discontinue this shortly

12. Repository We use the sPHENIX-collaboration organization in githubhttps://github.com/sPHENIX-Collaboration which owns all the sPHENIX repositories. Currently we have 26 repositories: sPHENIX-Collaboration/CAEN_VME sPHENIX-Collaboration/GenFit sPHENIX-Collaboration/HepMC3 sPHENIX-Collaboration/JETSCAPE sPHENIX-Collaboration/MVTX_FELIX sPHENIX-Collaboration/OnlMon sPHENIX-Collaboration/RDBC sPHENIX-Collaboration/Singularity sPHENIX-Collaboration/analysis sPHENIX-Collaboration/calibrations sPHENIX-Collaboration/coresoftware sPHENIX-Collaboration/digitzer_testcode sPHENIX-Collaboration/drs4 sPHENIX-Collaboration/gprof2dot sPHENIX-Collaboration/macros sPHENIX-Collaboration/macrosIDTF sPHENIX-Collaboration/online_distribution sPHENIX-Collaboration/opt_sphenix_scripts sPHENIX-Collaboration/phenixdaq_study sPHENIX-Collaboration/pythia6 sPHENIX-Collaboration/rapidjson sPHENIX-Collaboration/rcdaq sPHENIX-Collaboration/srs sPHENIX-Collaboration/tpc_dam sPHENIX-Collaboration/tutorials sPHENIX-Collaboration/utilities No private repositories – everything is world readable be mindful what you put out there Write access given on a repository basis Some repositories are forks where the original is itself a git repository (GenFit, HepMC3, JETSCAPE, gprof2dot, rapidjson)

13. Repositories you should be aware of sPHENIX-Collaboration/Singularity : Singularity Container for sPHENIX sPHENIX-Collaboration/analysis : user analysis software (write access for all collaborators) sPHENIX-Collaboration/calibrations : calibration files sPHENIX-Collaboration/coresoftware : our software sPHENIX-Collaboration/macros : ROOT macros to run sPHENIX-Collaboration/online_distribution : Event libraries Code changes in the coresoftware and macro repository use Pull Requests: You modify the code in your local git repository, push this to github and submit a pull request. The proposed changes get reviewed (informal) and if accepted (most of the time) the Pull Request gets merged into the software. The analysis repository does not use pull requests The size of the analysis repository will be a problem soon – need to come up with a solution We are setting up “continuous integration” which runs automatic checks on every Pull Request Jin has set up Jenkins: https://jenkins.io and is testing it And we do have a coding conventions: https://wiki.bnl.gov/sPHENIX/index.php/Codingconventions

14. The jobs consistently abort in the following step: Fun4AllServer::process_event processing SILICON_TRACKER process_event: cded to TOP/SILICON_TRACKER debugging • Ordered according to level of desperation • Print statements: enable verbosity to check where program dies, add • printouts in code, recompile, stare at screen, add more printouts, • recompile, stare at screen,… • Gdb (GNU Debugger): stops when segfault occurs, enables you to check • values of variables. For easier use compile without optimizer. Normally • cannot locate memory corruption • Valgrind: Runtime check: Checks for use of uninitialized variables or invalid read/writes (random memory locations) • Caveat: invalid read/write to initialized memory are not flagged (e.g. writing by one after array boundary which hits • the next data member of a class) • Insure: needs special compilation which adds fences around arrays, runs many hours per event • Adress sanitizer (under testing): ROOT5 bombs out during startup, not tested with ROOT6 Pre-emptive tools we have (your code should pass those, it’s just good coding practice to run those) Coverity: commercial static analyzer, we have a daily build with it Cppcheck: public domain static analyzer, runs fast but can pick up “obvious” array boundary violations Callgrind: profiling – if you are tired of waiting for your code to finish Instructions in our wiki https://wiki.bnl.gov/sPHENIX/index.php/Tools

15. Singularity • SL7 supports running “containers” (light weight virtual machines). • Singularity is the technology of choice – rcf provides an SL6 (previous OS version) • and SL7 image. • Makes us independent from the underlying OS – we can run anywhere where • Singularity is supported and get identical results • Proof of principle Thanks to Ross Corliss and Ron Soltz • With cvmfs this will enable us to run on the OSG • Ron will use this at LLNL • Frees racf to upgrade OS independently of experiments

16. ROOT6 the easy part ROOT 5.34 is from 2012, somewhat regular updates stopped end of 2015 More and more 3rd party software requires root6 (currently genfit, hepmc3) APIs largely identical, root versioning can be used to distinguish between root5/root6 CINT  Cling (piece of crap replaced by a piece of garbage) Small change in Makefile.am and configure.ac needed to compile our code with root5 and root6 (yes, they could not resist to add – undocumented - command line parameters to rootcling) And they invented a new file (*dict.pcm) which contains the info to serialize (store and read) the class which needs to be copied to the location of the shared libraries On the bright side – pyROOT is back

17. ROOT6 invasive changes • Cling the new command interpreter is VERY different from CINT • Cling actually compiles the macros using some patched clang  much more picky in • terms of macro errors • It needs all include files for classes being called on the cmd line, cryptic not intuitive • loading of shared libraries • With root versioning and patiently trying, root5 macros can be changed to work under • root5 and root6 (done for our macros in g4simulation) • Currently needs to include all our include directories (just $OFFLINE_MAIN/include • does not work). All possible paths added by sPHENIX setup script. Script to add • include paths from your install area exists. • We have identically named include files with different content in the prototype • packages which now have to be installed. This needs to be changed, one might be able • to hack around this but this approach is just a really bad idea • ROOT6 files are larger by 10%, ROOT5 cannot read them (at least I did not find a way). • But ROOT6 can read ROOT5 files

18. Upgrading • Geant4: our current version G4 10.02 is old, especially for the TPC we • want to use 10.04 which is better in reproducing the energy loss in gas • ROOT6 – the elephant: • nothing prevents us from switching right now (except that all your • macros will break) • Results root5/root6 identical (minor last digit diffs since roots Tmath • is just garbage) • Genfit: • needs ROOT6, small diffs between versions but needs quantifying • uses Eigen instead of TMatrix and should be faster • Hepmc: HepMC3 seems viable, much better than HepMC2 (real stl iterators), we need hepmc3 to get serious with jetscape Problem: We have no clear upgrade strategy – we have been talking about upgrading Geant4 for a year. Relying on voluntary checking just does not work Keeping many different version functional takes effort and manpower we just do not have

19. EIC software consortium update eRD20 Fun4All (and EICRoot from Alexander) is the only software which can run EIC detector simulations Got a preliminary physics list for EIC energies (which is coincidentally sPHENIX energies)

20. Changes in rcf • In the process of renaming themselves to SDCC (Scientific Data and Computing Center) • Reorganized with subgroups and respective group leaders (org chart next slide) • Moving towards general purpose facility – RHIC experiments are not consulted anymore • which leads to sub-optimal and more expensive implementations • So far this does not affect sPHENIX (effect on highmem jobs needs to be seen) • But it has a real impact on planning how to run sPHENIX - can we count on real time • racf resources? will they take on anything besides data archiving? • Major change in condor setup, no more “our” machines, just one condor pool. Not clear • how this will affect himem jobs of sPHENIX (might increase number of slots, or might not) All this being said – we do have a very good working relationship with rcf staff Single Signon coming next Tuesday!!!!

22. BNL internal review 5 minute presentation for each experiment (sPHENIX, STAR, Dune, Atlas, Belle, racf) Funding for all computing is funneled through CSI, physics needs its own voice, racf is not doing this Talk about pulling people out of groups to form a computing group All experiments spoke out strongly against this plan except STAR which promotes this idea

23. sPHENIX Software and Computing Review https://indico.bnl.gov/event/4728 Committee members: Jerome Lauret (BNL, STAR, co chair), Torre Wenaus(BNL, ATLAS, co chair) Christoph Paus (MIT, CMS), Brett Viren (BNL, Dune), Tony Wong (BNL, racf)

24. Charge to the Review Committee • The sPHENIX detector, currently under development, is designed to facilitate large acceptance, ultra-high rate measurements of fully reconstructed jets and high resolution spectroscopy of Upsilon states at the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory (BNL). The experiment is aimed at addressing scientific questions prioritized in the 2015 NSAC Long Range Plan and generally enhancing the physics reach afforded by the RHIC complex prior to the possible construction of an Electron Ion Collider (EIC). • The committee is charged to review the software and computing aspects of the project and recommend corrective actions in case deficiencies are found: • Are the resources required to transmit and store the data adequately understood? • Are the resources required to process the data to a form suitable for physics analyses adequately understood? • Is the plan for developing software processes and framework adequately understood?

25. Summary of our presentations: daq • Data Volume • Run 1: • Au+Au: 14.5 weeks ⋅ 60% RHIC uptime ⋅ 60% sPHENIX uptime  47 billion events, 1.7MB/evt 75PB • Run 2 and 4: • p+p, p+A: 22 weeks ⋅ 60% RHIC uptime ⋅ 80% sPHENIX uptime  96 billion events, 1.6MB/evt 143PB • Run 3 and 5: • Au+Au: 22 weeks ⋅ 60% RHIC uptime ⋅ 80% sPHENIX uptime  96 billion events, 2.3MB/evt 205PB Bandwidth (best week) Run 1: 109 Gb/s Run 2,4: 113 Gb/s 36 pairs of fibers from 1008  rcf, each pair can do 100Gb/s Run 3,5: 178 Gb/s more than enough bandwidth

26. Summary of our presentations: tracking • CMS track fit performance • 1ms per track (14.5 hits on track) • ATLAS tracking (ACTS Package), Preliminary estimate based on sPHENIX geometry • Track propagation – 0.5ms/track • Full RKF fit – 1-2 ms • Projection sPHENIX: • Min Bias 49 sec -> ~1.0 sec • Min Bias + PU 65 sec -> ~1.5 sec • Assume a conservative kalman filter approach with a fully optimized code implementation • MB events + pileup: • 5 sec/event for tracking • 5 sec/event for TPC clustering, calibration, 3D vertexing and other services • 5 sec/event for calorimeter and particle flow reconstruction • => 15 sec/event total projected event reconstruction time • Prepare to reconstruct 96 billion events in Year-3 • 96 000 000 000 ev / 3600 * 24 * 365 sec ~ 3000ev/sec • At 15 sec per event: • 45k equivalent-cores to reconstruct the data in the same year • 90k equivalent-cores for prompt reconstruction, reconstructing the data as they are taken Remember slide 3: racf has 57700 cores in total right now

27. Hpss HPSS disk cache Raw HPSS disk cache DST Buffer boxes Tape DST Disk cache Analysis Taxi Summary of our presentations: Data Reconstruction 110Gb/s Raw Disk cache 20PB (2 weeks) Calibration + Q/A reconstruction Conditions DB Raw data Disk cache size should be sufficiently large to buffer 2 weeks of data Reconstructed data disk cache should ideally keep all reconstructed output Online monitoring

28. Observations • The presentations were carefully prepared in a timely way and explicitly addressed the points in the • charge, and questions we had during the review were followed up quickly. This was much appreciated • by the committee. • The NPP ALD clarified that the resources the committee should be considering in the charge include • effort, hence the questions posed by the committee on FTEs. • The current S&C effort level is ~10 FTEs distributed over ~23 people. As students and postdocs ramp • up between now and startup, the anticipation is a ~23x increase in available effort. • We were told that the scale of the core ‘professional’ S&C effort that will be needed will be similar to • that of STAR (i.e. circa 10 people in the core team located at BNL, 6 of which are directly working on • software development). This would represent a different model from that of PHENIX while the number • of FTE of both major RHIC experiments (STAR and PHENIX) during their golden years were equivalent • as showed at past Program Advisory Committee (PAC) and Science and Technology (S&T) reviews, • emphasis was made on detector operation in PHENIX (engineers & professionals) with little on • computing, unlike STAR which allocated more workforce on computing (IT professionals) with a thinner • detector operation team as a result. A new relative apportionment of workforce within the sPHENIX • team to boost the S&C efforts would be required and the sPHENIX collaboration supports this direction.

29. Recommendations, general Clarify the physics requirements and how they map to computing resources and deliverables. This should be provided by the end of 2018. Develop a high level plan containing a timeline, milestones and effort level estimates (available and needed effort), by beginning of 2019. The plan should include activities through commissioning in 2022, and should have a cost and FTE profile indicating the BNL based FTEs and those drawn from other sources such as the collaboration at large. Develop in close collaboration with the RACF and NPP management a plan for sPHENIX compute and storage resources at BNL. Cost and ease of use trade offs should be important considerations. We encourage an active, transparent and open dialog helping sPHENIX establish their computing model. Plan a followup review; circa second quarter 2019 may be an appropriate interval. The review should be before CD2. Plan a duration of 1.5 days or so next time. The committee had very little time to prepare adequately for the closeout, and next time the review is likely to cover more material. A reasonable format would be to conclude the first day after completing the presentations and some committee discussion, ask any overnight questions, and spend the next morning on report writing before the closeout. We recommend to BNL management that the laboratory encourage and catalyze collaboration between CSI and experiments in the Physics Department. Crossdepartment project teams that aim to boost the capabilities and functionalities of experimental software (projects of broad interest and impact) are highly desirable. One example of a mechanism could be Laboratory Directed Research and Development funds support.

30. Recommendations Data acquisition and storage Quantify the uncertainties in TPC data volume for possible commissioning and zero suppression scenarios. Present these scenarios at the next review. Discuss with CAD to optimize the beam parameters by the end of 2018. 2.) John Haggerty is in contact with CAD about exactly this

31. Recommendations Data processing and simulation • Complete the resource spreadsheet shown during the review to include CPU, and provide the numbers to the committee within a week’s timefrom the delivery of this report. The table should include: • prompt reconstruction • reconstruction reprocessing • organized (taxi based) analysis • simulation • Develop a computing resource model taking into account cost/benefit considerations, experience from existing and prior experiments, resource requirements of the primary workflows, and the potential for leveraging resources outside BNL and HPCs. The model should be documented in early 2019, well in advance of the next review. • Actively engage with CSI to leverage their expertise and knowledge in CPU technologies as applicable. At the next review, summarize the outcome of the interactions and possibly present findings or recommendations issued from interacting with CSI. • Extend the full simulation software to support alignments, calibration uncertainties and full detector response in order to be ready to validate the coverage of all physics requirements. • Clarify the process of alignment, calibration and distortion corrections in the data reconstruction workflows (timeline, contingencies) and how contingencies would affect physics deliverables and production plans. • Demonstrate plans to exploit resources outside of BNL and evaluate the use of HPC resources for some of the needed workflows. • Include data challenge milestones in the plans for the precommissioning period, including effort needed.

32. Recommendations Framework and software development Determine a target memory level at which migrating to multithreading (MT) is necessary; i.e. if memory consumption per core is above this level, MT will be needed in order to reduce it. Evaluate the implications and effort involved in migrating to a MT based framework, to inform a decision on the migration with clear justifications for a MT or nonMT approach and present a summary at next review meeting. Consider designing the framework with support for a distributed computing approach in mind. 1) ATLAS ACTS available under https://gitlab.cern.ch/acts/acts-core