CyberShake Study 14.2 Technical Readiness Review

1. CyberShake Study 14.2 Technical Readiness Review

2. Study 14.2 Scientific Goals • Compare impact of velocity models on Los Angeles-area hazard maps with various velocity models • CVM-S4.26, BBP 1D, CVM-H 11.9, no GTL • Compare to CVM-S, CVM-H 11.9 with GTL • Investigate impact of GTL • Compare 1D reference model • Compare tomographic inversion results • 286 sites (10 km mesh + points of interest)

3. Study 14.2 Technical Goals • Run both SGT and post-processing workflows on Blue Waters • Plan to measure CyberShake application makespan • Equivalent to the makespan of all of the workflows • (All jobs complete) – (first workflow submitted) • Includes hazard curve calculation time • Includes system downtime, workflow stoppages • Will estimate time-to-solution by adding estimate of setup-time and analysis time. • Compare performance, queue times, results of GPU and CPU AWP-ODC-SGT

4. Performance Enhancements • New version of seismogram synthesis code to reduce read I/O • Reads in set of extracted SGTs • Synthesizes multiple RVs (using 5 in production) • Reduce number of subworkflows to 6 (from 8) • Fewer jobs, less queuing time • For CPU SGTs, increase core count • Each processor has ~64x50x50 chunk of grid points • For GPU SGTs, decrease processor count • Volume must be multiple of 20 in X and Y • 10 x 10 x 1 GPUs, regardless of volume

5. Proposed Study sites (286)

6. Study 14.2 Data Products • 2 CVM-S4.26 Los Angeles-area hazard models • 1 BBP 1D Los Angeles-area hazard model • 1 CVM-H 11.9, no GTL Los Angeles-area hazard model • Hazard curves for 286 sites x 4 conditions, at 3s, 5s, 10s • 1144 sets of 2-component SGTs • Seismograms for all ruptures (~470M) • Peak amplitudes in DB for 3s, 5s, 10s

7. Study 14.2 Notables • First CVM-S4.26 hazard models • First CVM-H, no GTL hazard model • First 1D hazard model • First study using AWP-SGT-GPU • First CyberShake Study using a single workflow on one system (Blue Waters)

8. Study 14.2 Parameters • 0.5 Hz, deterministic • 200 m spacing • CVMs • Vs min = 500 m/s • UCERF 2 • Graves & Pitarka (2010) rupture variations

9. Verification • 4 sites (USC, PAS, WNGC, SBSM) • AWP-SGT-CPU, CVM-S4.26 • AWP-SGT-GPU, CVM-S4.26 • AWP-SGT-CPU, BBP 1D • AWP-SGT-GPU, CVM-H 11.9, no GTL • Plotted with previously calculated curves

10. CVM-S4.26 (CPU)

11. CVM-H, no GTL (CPU)

12. Changes to SGT Software Stack • Velocity Mesh generation • Switched from 2 jobs (create, then merge) to 1 job • SGTs • AWP-ODC-SGT CPU v14.2 • Has wrapper because of issue with getting exit code back • AWP-ODC-SGT GPU v14.2 • Has wrapper to read in parameter file and construct command-line arguments • Nan Check • Always had NaN check for RWG SGTs, now for AWP SGTs also

13. Changes to PP Software Stack • Seismogram Synthesis / PSA Calculation • Modified to synthesize multiple seismograms per invocation • Will use 5 rupture variations per invocation • Reduces read I/O by factor of 5 • Needed to avoid congestion protection events • All codes tagged in SVN before study begins

14. Changes to Workflows • Changed workflow hierarchy • 1 integrated workflow per site, per • Added ability to select SGT core count dynamically • Put volume creation job into top-level workflow to reduce hierarchy to 2 levels • Reduced number of post-processing sub-workflows to 6 • Fewer jobs in queue • Will not keep job output if job succeeds • Reduce size of workflow logs

15. Workflow Hierarchy Integrated Workflow (1 per model per site) PreCVM(creates volume) SGT Workflow Generate SGT Workflow More details on next slide PP Pre Workflow PP subwf 0 PP subwf 1 PP subwf5 … DB workflow

16. 68000 68000 6

17. Distributed Processing • Cron job on shock.usc.edu creates/plans/runs full workflows • Pegasus 4.4, from Git repository • Condor 8.0.3 • Globus 5.0.4 • Jobs submitted to Blue Waters via GRAM • Results staged back to shock, DB populated, curves generated • Alternate CPU and GPU workflows for best queue performance

18. Computational Requirements • Computational time: 275K node-hrs • SGT Computational time: 180K node-hrs • CPU: 150 node-hrs/site x 286 sites x 2 models = 86K node-hrs (XE, 32 cores/node) • GPU: 90 node hrs/site x 286 sites x 2 models = 52K node-hrs (XK) • Study 13.4 had 29% overrun on SGTs • PP Computational time: 95K node-hrs • 60 node-hrs/site x 286 sites x 4 models= 70K node-hrs (XE, 32 cores/node) • Study 13.4 had 35% overrun on PP • Current allocation has 3.0M node-hrs remaining

19. Blue Waters Storage Requirements • Planned unpurged disk usage: 45 TB • SGTs: 40 GB/site x 286 sites x 4 models= 45 TB, archived on Blue Waters • Planned purged disk usage: 783 TB • Seismograms: 11 GB/site x 286 sites x 4 models= 12.3 TB, staged back to SCEC • PSA files: 0.2 GB/site x 286 sites x 4 models= 0.2 TB, staged back to SCEC • Temporary: 690 GB/site x 286 sites x 4 models= 771 TB

20. SCEC Storage Requirements • Planned archival disk usage: 12.5 TB • Seismograms: 12.3 TB (scec-04 has 19 TB) • PSA files: 0.2 TB (scec-04) • Curves, disagg, reports: 93 GB (99% reports) • Planned database usage: 210 GB • 3 rows/rupture variation x 410K rupture variations/site x 286 sites x 4 models = 1.4B rows • 1.4B rows x 151 bytes/row = 210 GB (880 GB free) • Planned temporary disk usage: 5.5 TB • Workflow logs: 5.5 TB – possibly smaller, not saving all output anymore (scec-02 has 12 TB free)

21. Metrics Gathering • Monitord for workflow metrics • Will run after workflows have completed • Python scripts • Used to obtain some of the standard CyberShake metrics for comparison • Cronjob on Blue Waters • Core usage over time • Jobs running and idle counts • Will use start and end of workflow logs to perform makespan measurement

22. Estimated Duration • Limiting factors: • Queue time • Especially for XK nodes, could be substantial percentage of run time • Blue Waters -> SCEC transfer • If Blue Waters throughput is very high, transfer could be bottleneck • With queues, estimated completion is 4 weeks • 1 hazard map/week • Requires average of 410 nodes • 603 nodes averaged during Study 13.4 • With a reservation, completion depends on the reservation size

23. Personnel Support • Scientists • Tom Jordan, Kim Olsen, Rob Graves • Technical Lead • Scott Callaghan • SGT code support • EfecanPoyraz, Yifeng Cui • Job Submission / Run Monitoring • Scott Callaghan, David Gill, Heming Xu, Phil Maechling • NCSA Support • Omar Padron, Tim Bouvet • Workflow Support • Karan Vahi, Gideon Juve

24. Risks • Queue times on Blue Waters • In tests, at times GPU queue times have been > 1 day • Congestion protection events • If triggered consistently, will either need to throttle post-processing or suspend run until improvements are developed

25. Thanks for your time!