Model Coupling Toolkit: Recent Developments and Future P lans

1. Model Coupling Toolkit: Recent Developments and Future Plans Robert Jacob Argonne National Laboratory Second Workshop on Coupling Technologies for Earth System Modeling, NCAR, February 20-22, 2013.

2. About Argonne • Founded in 1943, designated a national laboratory in 1946 • Managed by The University of Chicago for the U.S. Department of Energy • More than 2,900 employees and 5,000+ facility users • About $475M/year budget • 1,500-acre, wooded site in DuPage County, Illinois • Broad science portfolio • Numerous sponsors

3. MCT Philosophy: Model coupling vs. “the coupler” • MCT is not a coupler • Instead, MCT provides datatypes/methods you add to models (and a separate coupler or driver if you want one) to make the models “couple-able”. • Must also follow a few programming standards (suggested but not required for using MCT) • Separate “initialization” and “run” methods. • Do not use MPI_COMM_WORLD everywhere. • Avoid global data types

4. Model Coupling Toolkit: History • Pre-History: • 1996 parallel coupler for Fast Ocean Atmosphere Model (Jacob, UW-Madison) • 1998 Physical-space Statistical Analysis System (Larson, NASA) • U.S. Department of Energy ACPI Avante-Garde project (2000-2001) • First work on parallelizing CCSM coupler (cpl5->cpl6) • MCT 1.0 • DOE Scientific Discovery through Advanced Computing (SciDAC) (2001-2011) • MCT 2.0 – MCT 2.7.3 • CCSM3, CCSM4, CESM1. • DOE Climate Science for a Sustainable Energy Future (CSSEF) (2011 – 2015) • MCT 2.8 • Next generation MCT All DOE support is from the climate modeling program in the Office of Biological and Environmental Research (BER) in the Office of Science.

5. MCT Architecture { High-level MCT classes Low-level MCT classes Message-Passing Environment Utilities (MPEU)

6. mpeu (message passing environment utilities) Developed by the NASA DAO, and extended by MCT developers, mpeu provides the following services to Fortran90 MPI applications: • Support for basic derived types (List, String) on which low-level classes in MCT are built And…. • F90 module-style access to MPI • Support for multiprocessor stdoutandstderr • Error handling / shutdown • Support for namelist replacement “resource files” • Run-time flow tracing • Timing/Load balance measurements

7. MCT Low-level Classes • Coupled model registry (describe how many models are coupled (no limit)) • MctWorld • Multi-field data storage (hold data being transferred (any amount)) • AttrVect • Domain decomposition (any grid, any decomposition) • GlobalSegMap • Intercomponent parallel data transfer scheduler (between two GSMaps) • Router • Intercomponent parallel data transfer (For a Router and an Av) • Transfer • Intracomponent parallel data redistribution (for an AV and a GSMap of the same grid) • Rearranger

8. MCT High-Level Classes and Modules • Combining sources from two or more models • Merge • Communication methods for MCT datatypes • AccumulatorComms • AttrVectComms • GeneralGridComms • GlobalSegMapComms • SparseMatrixComms • Interpolation (sparse) matrix object • SparseMatrix • Sparse matrix – Attribute Vector multiply (for interpolation) • MatAttrVectMult • Physical Grid Description • GeneralGrid • Time averaging and accumulation support • Accumulator • Masked/unmasked spatial integrals and averages • SpatialIntegral

9. Typical MCT Use: ATM (M nodes) CPL (N nodes) OCN (P nodes) Call MCT World Define GlobalSegMaps Define AttrVects Define Routers Define Accumulators Read Matrix elements Call MCT World Define GlobalSegMap Define AttrVect Define Router Call MCT World Define GlobalSegMap Define AttrVect Define Router Initialization Read Atmosphere Data DO WORK Read Ocean Data DO WORK MCT_Send(AtrVect, Router) MCT_Recv(AAtrVect, ARouter) MCT_Recv(OAtrVectin, ORouter) MCT_AvMatVectMult(AAtrVect, SparseMatrix, OAtrVectout) Compute Fluxes MCT_Send(AAtrVect, ARouter) MCT_Send(OAtrVect, ORouter) MCT_Send(AtrVect, Router) MCT_Recv(AtrVect, Router) MCT_Recv(AtrVect, Router)

10. More on MCT Use • The user must: • Know how data is layed out on processors. • Describe decomposition to MCT with a GSMap • Points are uniquely numbered globally • Copy local data in to an MCT Attribute vector. • Copy in either memory order or global index order • Calculate Interpolation weights (with SCRIP or ESMF Regridder) • Read in interpolation weights (to root node) • MCT can: • Derive communication tables between decompositions (using indices) • Do all parallel data communication necessary for interpolation, gather, scatter. • Minimizes sizes of data transferred.

11. MCT Users • “IPCC-class” production coupled model: The NSF/DOE Community Earth System Model • MCT is the default coupling method in CCSM4 and CESM1 • MCT datatypes always used in top level coupler driver. • MCT methods/datatypes are default for driver-component communication. • All AR5 simulations by CCSM4/CESM1 are using MCT. • Other academic coupled systems: • COAMPS/ROMS - Hurricanes • ROMS/Swan - coastal oceanography • WRF/ROMS - Hurricanes • OASIS3 – MCT • See next talk

12. CESM using cpl7/MCT can scale to 100K cores.

13. MCT Recent history • 01/06/2010: MCT 2.7.0 released in CCSM4 • Limted used of OpenMP • 02/28/2010: MCT 2.7.1 released in CESM1 • 11/30/2010: MCT 2.7.2 released in CESM1.0.3 (CW2010 in Toulouse, France. December, 2010) • 01/25/2011: MCT 2.7.3 add debugging option to configure (2011: Some divergence between Argonne and NCAR MCT repositories) • 02/07/2012: MCT 2.7.4 update autoconf build to latest version (Jim Edwards)

14. MCT More Recent history • MCT 2.8.0 - Released April 30, 2012 (first standalone release since 2.6)! • Merged differences in Argonne and NCAR MCT repos. • New datatype in AttributeVector to speed up copies (thanks to Bill Sacks, NCAR) • ANL and NCAR repos in sync! • 07/12/12 - MCT 2.8.1 Convert Argonne repository to git • Repository now world readable. Copy on github.com • Full 10+ year history MCT development converted. • Github provides SVN interface allowing CESM to pull directly. Eliminate duplicate repo at NCAR. git clone http:git.mcs.anl.gov/MCT.git

15. Continuing to improve cpl7/MCT performance at scale. • Slow initialization time for high-resolution, high-processor count cases. • 1/8th degree runs on Intrepid were taking over 1.5 hours to initialize. • Traced to initialization of MCT’s Rearranger (equal to 2 Routers) which moves data between overlapping decompositions. Thanks Tony!

16. MCT More Recent history • 09/12/12 - MCT 2.8.2 • Includes fix for slow Router init. • Released in CESM 1.1 • Not released separately. • 12/19/12 – MCT 2.8.3 Current Version • Public release • All of above changes plus some minor compiler fixes

17. CSSEF research demands on coupling • Dynamical Adaptive Atmospheric Dynamics • Grid points are created and destroyed on a coupler processor • Changing cell sizes for just one grid within coupler will require online calculation of new interpolation weights • Which requires more information about both grids then currently in coupler. • Development of MPAS-Ocean • Need to retain information about unstructured grids for interpolation weight calculation. • Resiliency and Scaling • Dynamic load balancing and resilient computing means points could move from processor to processor. • Millions of threads and small per-core memory means need more parallelism and optimize for low-memory

18. Solution: Re-Implement MCT data model with MOAB • MOAB = Mesh Oriented dAtaBase • A database for mesh (structured and unstructured) and field data associated with mesh • Tuned for memory efficiency first, speed a close second • Serial, parallel look very similar, parallel data constructs imbedded in MOAB interface • http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB • Developed under DOE SciDAC program • Includes parallel I/O and visualization capabilities. • Included in nuclear engineering exascale co-design center.

19. MOAB is already used in other projects, notably DOE-funded cryosphere modeling and nuclear reactor simulation. Like MCT, it is “battle tested”. Ice Sheet bed Klystron Mesh

20. MOAB Data Model • 4 fundamental “types”: • Entity: fine-grained entities in grid (vertex, tri, hex)‏ • Supported types: vertex, edge, tri, quad, polygon, tet, prism, pyramid, hex, septahedron, polyhedron • Mostly unstructured, though can represent structured (leveraging work done with ParVis). • Flexible in representing intermediate-dimension entities (internal edges/faces) • Entity Set: arbitrary set of entities & other sets • Parent/child relations, for embedded graphs between sets • Interface: object on which interface functions are called and through which other data are obtained • Tag: named datum annotated to Entitys, Entity Sets, Interface • Instances accessed using opaque (type-less) “handles” • MOAB is a C++ library. Fortran interface is iMesh

21. MOAB Data Model illustrated

22. Review: MCT Classes Mesh Index-space domain decomposition Fields MOAB provides different class structures that define mesh, fields, and domain decomposition

23. MCTWorld (Legacy) type MCTWorld integer :: MCT_comm integer :: ncomps integer :: mygrank integer,dimension(:),pointer :: nprocspid integer,dimension(:,:),pointer :: idGprocid end type MCTWorld • Lightweight component model registry that stores the coupled-system-wide MPI global communicator (and local PE rank on it), number of component models, number of PEs in each component, and global PE rank translation table • Registry methods: • Create/destroy - init()/clean() • Query: # components, components’ root PE ranks, rank translation

24. type MCTWorld integer :: MCT_comm integer :: ncomps integer :: mygrank iMesh_Instance :: mesh integer,dimension(:),pointer :: nprocspid integer,dimension(:,:),pointer :: idGprocid end type MCTWorld MCTWorld(MOAB) • Sole extension to the datatype is the MOAB mesh instance • MCTWorld was conceived as a "lightweight registry" that served as a directory service for intercomponent communications. Addition of an MOAB instance to it makes it considerably heavier, but converts it into a full-blown registry for coupling purposes.

25. AttrVect (Legacy MCT) type AttrVect type(List) :: iList type(List) :: rList integer,dimension(:,:),pointer :: iAttr real(FP) ,dimension(:,:),pointer :: rAttr end type AttrVect • Stores pointwise collections of REAL (INTEGER) fields, or attributes, indexible by string tags in iList (iList) • Key methods: • Create/destroy: init(), clean() • Query: length - lsize(), # REAL/INTEGER attributes - nIAttr()/nRAttr(), names of attributes • Manipulate: copy(), zero(), append attributes, Import/Export indivudual attributes, sorting,, cross-indexing of attributes

26. AttrVect(MOAB) type AttrVect type(List) :: iList type(List) :: rList iBase_TagHandle,dimension(:),pointer :: itagh iBase_TagHandle,dimension(:),pointer :: rtagh iBase_EntityHandle,dimension(:),pointer :: enths end type AttrVect • Built using Fortran interface to MOAB (iMesh) • INTEGER/REAL attribute lists retained • Natural equivalence between “attribute” and “tag” • Attributes now stored contiguously and referenced by a handle iBase_TagHandle (implemented as an integer) • Mesh entities referenced by iBase_EntityHandles

27. iMesh-AttrVect test program ! Initialize MCT (Default 3-D--but empty--iMesh instance created: call MCTWorld_init(1, MPI_COMM_WORLD, comm1, 1) ! Initialize MCT AttrVect: call AttrVect_init(av1, rList=‘field1:field2’, & lsize=avsize) ! Query embedded iMesh instance to determine dimensionality: call iMesh_getGeometricDimension(%VAL(ThisMCTWorld%mesh), & geom_dim, ier) ! iMesh query function on the new Av tag handle call iMesh_getTagName(%VAL(ThisMCTWorld%mesh), & %VAL(av1%rtagh(1)) , & tagname, ier, %VAL(10)) Other AttrVect methods from previous slide also available as-is

28. MCT on Mira - Argonne’s BlueGene/G • 48 racks • 1024 nodes per rack • 1.6 Ghz 16-2ay core processor and 16 GB RAM per node • 348 I/O nodes • 240 GB/s 35PB Storage • 768K cores • 768 TB Ram • 10PF peak

29. CESM with cpl7/MCT on BG/Q Status • Latest development version of CESM compiles and runs on BG/Q • Compare: 7.4 years/day on 512 BG/P nodes (2048 cores; mixed). • Compiler bug encountered and patched! • CESM is mixed-mode but currently any threading slows down the model.

30. Additional MCT development • New Features to aid in debugging Router times • GSMap and MCTWorld print(). • Print contents to ascii file for later reading • Router init internal timers • Invoked with optional string argument to Router init. • RouterTest.F90 - test program which reads in output GSMap and MCTWorld info and builds a Router. • Will build on same number of procs and same decomposition as original model. • On branch but not yet released • Next: Conversion from F90 to F95 (and later F2003)

31. MCT: To be continued… www.mcs.anl.gov/mct