Chombo: An HPC Toolkit for Adaptive Mesh Refinement Applications Brian Van Straalen Applied Numerical Algorithms Group

1. Chombo: An HPC Toolkit for Adaptive Mesh Refinement Applications Brian Van StraalenApplied Numerical Algorithms Group Lead Software Developer for Chombo Building Community Codes for Effective Scientific Research on HPC Platforms, Sept. 6-7, 2012, University of Chicago

2. Chombo: AMR Software Framework Goal: to support a wide variety of applications that use AMR by means of a common software framework. Refactoring of BoxLib to better enable general usage patterns, such as support for embedded boundary representations of complex geometries. Approach: Mixed-language programming: C++ for high-level abstractions, Fortran for calculations on rectangular patches. Re-useable components, based on mapping of mathematical abstractions to classes. Components are assembled in different ways to implement different applications capabilities. Layered architecture, that hides different levels of detail behind interfaces. Significant effort expended in maintaining professional software development team responsive to a variety of users. Status: Chombo 3.1 Open Source Release – March 9, 2012.

3. Layered Library Framework Multi-physics code development places a premium on the availability of a diverse and agile software toolset that enables experimentation. We accomplish this with a software architecture made up of reusable tested components organized into layers. • Layer 4: Complete parallel applications. AMRSelfGravity, AMRMHD, AMRINS, EBAMRINS, AMRCharm. 100,000 lines of code. • Layer 3: Solver libraries - geometric multigrid solvers on unions of rectangles, AMR hierarchies; hyperbolic solvers; AMR time stepping. 70,000 lines of code • Layer 2: Tools for managing interactions between different levels of refinement in an AMR calculation - interpolation, averaging operators, coarse-fine boundary conditions. 50,000 lines of code. • Layer 1: Data and operations on unions of rectangles - set calculus, rectangular array library (with interface to Fortran). Data on unions of rectangles, with SPMD parallelism implemented by distributing boxes to processors. Load balancing tools. 80,000 lines of code. • Utility Layer: Code instrumentation, interoperability libraries - API for HDF5 I/O, AMR data alias. Performance and debugging tools. 20,000 lines of code

4. Lower Triangular Dependency • Dependency Structure Matrix • Lower triangular implies layering • Avoids Loops. • Outside contributors add value at the bottom and the top mostly. • Different kinds of experts • More true triangular structure makes parallel development more effective. • Framework and Util still seems to require a guru. • Stronger interfaces essential at lower levels • Lower levels require the most advanced computer scientists • The middle levels (domain, etc) are very mathematical • The application levels are the most physics-oriented

5. Current Chombo Applications Core AMR capabilities • MHD for tokamaks using allspeed projection methods (R. Samtaney, PPPL) • Cosmology: compressible CFD + particles (F. Miniati, ETH) • Space plasmas: compressible CFD + electromagnetic, kinetic effects (G. Zank, UA-Huntsville) • Astrophysical MHD turbulence (C. McKee / R. Klein, UCB) Embedded Boundary Methods • SF Bay and Delta Hydrology modelling – shallow water (CA Department of Water Resources) • Plasma-wakefield accelerators – compressible viscous flow (W. Leemans, LBNL) • Gas turbines - allspeed projection methods (CCSE) • Blood flow in cerebral arteries – fluid / solid coupling (UCB ParLab project) • Nuclear reactor safety - entrainment models for hydrogen combustion (D. Calhoun, CEA-Saclay) • Complex fluids at the microscale (UNC / UCD / LBNL multiscale math project; LBNL EFRC) • Conjugate heat transfer in nuclear reactors (LBNL LDRD). Mapped Multiblock Methods • Type II Supernovae – compressible flow (S. Woosley, UCSC, LLNL) • 4D gyrokinetic models of tokamak edge plasmas (LLNL) • Land ice model for climate simulation – Stokesean solid (LANL) • Atmospheric models for climate simulation – low-Mach number CFD (C. Jablonowski, U. Mich.)

6. ANAG Software Engineering Practices Source code control Subversion repository Release branches Nightly regression testing 4 machines, 24/7 allocation Large configuration space Doxygen hypertext documentation Chombo User’s Guide Easy to port and modify Build based on just GNU make and perl Few external library dependencies (lapack, HDF5) Third-party capabilities can be used, but are not required • VisIt Integrated with GDB debugger • Software design process • Design Document • Example application • Unit tests • Chombo Coding Standards • Chombo users’ mailing list • Bug reporting • Community input

7. Performance Work Message Agglomeration Messaging overlapping Cached message patterns Copier Copier manipulations Space-filling curve ordering Morton-ordering Maximize locality Exploit sorting Moving Window Optimization Multigrid Specific “Lazy” Relaxation Residual-correction formulation Re-use of temporaries Coarse-Fine optimized stencils Reduction of norms vs. norms of reductions • Arena heap management • Manual loop fusion • Edge-trimming • EB-Specific • Stop using N-color relaxation • Bulk stencil operations • Solver re-use • Measurement-based load balancing

8. Chombo Performance Tools Chombo Timers Instrumented source code Portable Parallel or serial profiling Hardware counters Memory Tracking Memory leak detection Memory usage Parallel Debugging registerDebugger() attachDebugger() Event-based diagnostics

9. Chombo: Still Pretty Much a Cathedral Brian Van StraalenApplied Numerical Algorithms Group Building Community Codes for Effective Scientific Research on HPC Platforms, Sept. 6-7, 2012, University of Chicago

10. “Cathedral and the Bazaar”, Eric S. Raymond • The Cathedral model • Code is available with each software release • Development between releases is restricted to an exclusive group of software developers. • GNU Emacs and GCC are presented as examples. • Central control models • The Bazaar model • Code is developed over the Internet in view of the public. • Raymond credits Linus Torvalds, leader of the Linux kernel project, as the inventor of this process. • Distributed control models

11. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. • In this first rule we are more like a bazaar. • It is amazing how deep users will penetrate into your source code. • They WILL read your comments and be bummed out when they are out-dated or inaccurate. • Debugging is parallelizable. • Bugs also seem to be amazingly independent. • Brook’s Law (well, a lemma of it I guess) • Users are quite good at building a specific test code that demonstrates a bug. • Sometimes they need encouragement/instructions • Users are less good at coming up with good bug fixes • They develop workarounds and hacks

12. Scientific Codes must be reproducible • With the parameters in your paper, and access to your code, I should get the same answers. • Requires version control • “Improvements” that violate this property compromise the entire enterprise. • Constant regression testing • complete logging works wonders. • Mining regression output from months earlier is a great investigative tool. • What about disciplines that make no claim to reproducibility? • What is Wikipedia good at ?

13. Object-Oriented Programming Helps • Strong interfaces and encapsulation (enforced by the language or build system) enables community participation. • Users can try derived classes and get their code running without too much direct hand-holding. • Open-source means they can change interfaces locally, then talk to us. • Design-by-Contract (DbC) • C++ classes are good for both abstract datatypes and interface definitions. • conditions and invariants are managed in Chombo with ch_assert( ).

14. Open Source Benefits • Nobody can pull the plug on you. • You have the source code, free to use and modify, in perpetuity. • That includes me • You don’t have to pay • But you might be asked to help generate funding • You pay with your time and attention and what you give back. • Stakeholders are power (if you can figure out how to tap it) • Not all stakeholders are equivalent • User count has not been as helpful as vocal collaborators

15. }