Model Coupling Environmental Library

1. Model Coupling Environmental Library

2. Goals • Develop a framework where geophysical models can be easily coupled together • Work across multiple platforms, operating systems • Simple to use • Minimal API • Easy to add functionality • Minimally intrusive into existing code • Work for both one and two way coupling • Flexible coupling scheduling • “Reasonably” high performance

3. Target User Base • MCEL was designed to be used at the DoD research laboratories and operational centers • Oceanographers and meteorologists, not computer scientists. Simplicity over functionality! • FORTRAN 77 is a modern language • MCEL incorporates a simple FORTRAN and C API • 29 Total commands • OOPs languages can access MCEL directly • Typically requires about 100 additional lines of code to legacy model to couple using MCEL

4. Current Approach • Server/client approach • Centralized server(s) stores coupling information • Clients (numerical models) store and retrieve data from server • Data passes through filter(s) between server and client • Filters alter data into ready to use form for clients • Interpolation, data combination, physics… • Filters not tied to any models, at most tied to numerics/physics – Reusable • Server archives coupling data, bunches of netCDF files which can be relocated and restarted • Dedicated thread to file output, run at a lower priority not to slow services • Once data is stored, it CAN be scrubbed from memory, scrubbing handled by dedicated thread

5. Typical Example Wave Atmospheric Winds Atmospheric Atmospheric Energy xfer Atmospheric Waves Wind/Wave Interpolation SST Winds SST Server Interpolation Interpolation SST Waves Rad Stress Ocean Surface Forcing

6. Communication Details • Communication handled through CORBA • Cross platform • Free • Multi language support (C++, Java, Python …) • MCEL API has FORTRAN/C bindings, object hiding • Security through SSL • Passive waits available • Over-subscription of CPUs • Components can come and go as needed • Built in multithreaded support

7. Communication Infrastructure

8. Component Details • Currently, all processes initiated by the components • Models are required to “register” themselves with the coupling framework • Define grid and variable information • Grids can be either one, two or three dimensional: regular, structured or unstructured: Lat/Long or Cartesian Coordinates • Variables can be one of the standard MPI types • Models then set up filter tree • Models periodically store data to server • Models periodically request data from filter(s) • Models use standard time stamp to mark all data

9. Component Details Cont. • Components are all generic, coupling is performed through variable name matching • Components can use any parallelization scheme • MPI, OpenMP, Vector, pthreads… • Each component process/thread can communicate independently or through the master • Component N to M communication results in N-1-M communication. However, 1 is multithreaded and work is ongoing to fix this • API allows data to be pre-fetched in a non-blocking way • Concept: On cluster of SMP machines about looking in a local cache first before going to the server

10. Data Shipment • Upon model registration, grid and variable information is passed around between components, filters and servers • During operation, only the raw data and variable/temporal information is passed around • Unless a model has had a grid change, then new grid info is distributed • Filter outputs available to be shared if still in local cache • Caching of filter inputs being incorporated • MPI application requesting the same data for each task

11. Filter Details • Filters allow the encapsulation of physics or numerics into a reusable package such as interpolation • Three basic functions, initialize, evaluate and shutdown • Inherited off virtual C++ base class which include all the basic MCEL communication infrastructure • Just supply your filter numerics in any language • Filters created in “factories” and can created and destroyed as required • Filters are multithreaded • Filters can have multiple parents and multiple children • Compute once, share to many • Cache flushed on subsequent calls

12. Performance Details • Results are for a 512x512 grid. If interpolation is used, going to an unstructured grid with 40K points • MCEL storage and retrievals typically on the order of five milliseconds (No filters used) for a single process on the same node as the server • MCEL storage and retrievals typically on the order of ten milliseconds (No filters used) for a single process on a different node as the server (Compaq or IBM) • MCEL storage typically on the order of thirty milliseconds (No filters used) for a 32 process on different nodes as the server (Compaq or IBM) • MCEL retrieval time through an interpolation filter for a 32 process parallel job requires typically 0.1 to 0.2 seconds

13. ADCIRC ADH COAMPS HYCOM LSOM NCOM REF/DIF STWAVE SWAN WAM WaveWatch WRF Models Currently MCEL Aware

14. Closing • MCEL designed with legacy codes in mind and to be as easy to use as possible • MCEL reasonable for coupling every timestep at the most • Excessive overhead for inner-loop coupling • MCEL is in alpha release • MCEL is publicly available via anonymous CVS • :pserver:anonymous@attica.ssc.usm.edu:/home/matt/CVS • MCEL tutorial available at the DoD HPCMO UGC meeting in June