Coupled Domains in Coastal Ocean Simulations using MPIg on the NGS

1. Coupled Domains in Coastal Ocean Simulations using MPIg on the NGS Stephen Pickles*, Mike Ashworth*, Jason Holt# *CSED, STFC Daresbury Laboratory #Proudman Oceanographic Laboratory

2. Summary We describe how regional domains in coastal ocean simulations can be coupled through the periodic exchange of data along their shared boundaries, and explain why it is feasible to distribute such simulations over a wide-area Grid. We test this idea in a production coastal ocean modelling code using MPIg on the NGS, and quantify the performance penalty of cross-site runs.

3. Global Coastal Ocean Modeling System • GCOMS • NERC e-Science Project • POLCOMS (physics) • Finite differences, regular lat/long grid, Fortran+MPI, from POL • ERSEM (ecosystem) • from PML • Need higher resolution for coastal/shelf seas than for deep ocean • GCOMS domains have 0.1 deg grid-spacing, 30 vertical levels • 2D domain decomposition S,T,u,v,w, SPM, Kh, Kv POLCOMS ERSEM

4. Inter-domain communications • GCOMS domains normally run independently, typically on 16-512 processors each • Idea: run several neighbouring domains in parallel, coupled though exchange of boundary data • temperature, salinity, velocity, elevations • instead of reading forcing data from disk • Volume of data exchanged between domains is then much less than intra-domain halo exchanges • This makes it feasible to place different domains on different hosts, and use a grid-enabled flavour of MPIg for inter-domain communications • Despite much higher latency and lower bandwidth Coastal ocean divided into ~50 “domains”

5. Cross-site runs using MPIg on the NGS • Domains are placed on different hosts • We used the NGS nodes at Manchester and Leeds • No others had MPIg set up for Fortran • MPIg used for communication • No source code changes for MPIg • Load balancing across domains/hosts is done manually • 2 smallish domains of similar size • Off the West African coast • Inter-domain boundary is short (~30 grid points)

6. Performance

8. Discussion • On a single host, the overhead of coupling domains is small • The overhead of cross-site runs is significant, but not preclusive • Could still be viable if couldn’t get enough resource on a single host • Important not to split a domain across hosts • Load balancing between domains is done manually • Forcing data at domain boundaries is time-interpolated • 5 minutes (model time) for temperature and salinity • But every 20 seconds for velocities and elevations • This is the rate-limiting step for inter-domain communications! • Have validated inter-domain comms on up to 6 domains • But have only measured performance on 2 domains • Our method will not scale to many domains • Have performed 5-year hindcast run on HECToR to study effect of inter-domain comms on physics

9. Experiences of MPI and MPIg • MPI • Want standard MPI ways to probe underlying topology • Increasingly important for multi-core, not just for metacomputing • One for MPI3? • MPI_ABORT did not abort on the NGS! • MPIg • No signs of a general release • No issues encountered with correctness • But installation and configuration was difficult • Need full commitment from systems administrator • When it went wrong, the symptom was invariably a hang • Difficult to diagnose • MPIg jobs are run “interactively” using globus-run (or a thin wrapper) • harder to prepare scripts for unattended runs

10. Experiences of HARC • HARC (Highly Available Resource Co-allocator) • Needed to get resources at both sites • Generally worked well • Not all sites ran latest version • Had to be alert for time-zone bugs around summer time • Fixed in later release • Little user documentation of HARC client • Idiosyncratic time format • Like RFC, but not quite

11. MPIg for production work? • There is more that can go wrong using MPIg than vanilla MPI • All processes on each host must be able to communicate • Multiple layers (vendor MPI, MPIg, Globus, compilers) must be compatible • Problem resolution is much harder • Load balancing between domains on possibly heterogeneous hosts • Today, it is easier for us to find a single host with sufficient resources for a given production run, than it is to find a set of MPIg-enabled hosts with equivalent aggregate resources • Consequently, we do not use MPIg in production

12. Acknowledgments • This work was supported by • NERC Grant NE/C516001/1 (GCOMS) • UK National Grid Service • Special thanks to:- • Crispian Batstone • for starting this work • James Harle • for the input data used in these experiments • Nicholas Karonis and Brian Toonen • for helpful advice about MPIg • Matt Ford and Jason Lander • for persevering with MPIg on the NGS

13. Questions?