What is ESMF and what does it mean to adopt it?

1. What is ESMF and what does it mean to adopt it? NCAR/LANL CCSM Climate Data Assimilation GFDL FMS Suite NASA GMAO Analysis GMAO Seasonal Forecast Weather MITgcm 3rd ESMF Community Meeting Cecelia DeLuca cdeluca@ucar.du Nancy Collins nancy@ucar.edu Jon Wolfe jwolfe@ucar.edu NCEP Forecast

2. Outline • What is ESMF? • ESMF users • A closer look at ESMF • Adoption strategies • A couple suggestions for how to get started

3. ESMF Architecture • ESMF provides an environment for assembling geophysical components into applications, with support for ensembles and hierarchies. • ESMF provides a toolkit that components use to • increase interoperability • improve performance portability • abstract common services

4. Hierarchies and Ensembles ESMF encourages applications to be assembled hierarchically Coupling interfaces are standard at each layer Components can be used in different contexts ESMF supports ensembles with multiple instances of components running sequentially (and soon, concurrently) Ensemble Forecast Seasonal Forecast assim_atm sea ice assim_atm ocean assim_atm assim_atm assim atmland atm land coupler physics dycore

5. ESMF Data Classes Model data is contained in a hierarchy of multi-use classes. The user can reference a Fortran array to an Array or Field, or retrieve a Fortran array out of an Array or Field. • Array – holds a Fortran array (with other info, such as halo size) • Field – holds an Array, an associated Grid, and metadata • Bundle – collection of Fields on the same Grid • State – contains States, Bundles, Fields, and/or Arrays • Component – associated with an Import and Export State

6. ESMF Utilities • Communication libraries • Regridding library (parallelized, on-line SCRIP) • Message logging • Calendar management – clocks, alarms, calendars • Configuration (replaces namelists) • IO – currently netCDF • Performance profilng (not implemented yet)

7. Outline • What is ESMF? • ESMF users • A closer look at ESMF • Adoption strategies • A couple suggestions for how to get started

8. What kind of user are you? Package code to run many places under many systems, support, documentation Provide components to others without large overhead Provider Support routines, utilities, communication code, leverage code developed elsewhere Full framework, inter-platform, inter-institution solutions User Individual/Small Group Large Group/Institution

9. What do you get from ESMF? • Different running configurations • Standalone • Coupled to data only (dead components) • Coupled to live components • Strategy for structuring complicated applications • Different communication strategies • Different data decompositions • Utilities

10. What do you get (cont.) • Standard component interfaces • Coupler-building utilities

11. Outline • What is ESMF? • ESMF users • A closer look at ESMF • Adoption strategies • A couple suggestions for how to get started

12. ESMF Superstructure Classes • Gridded component • Models, data assimilation systems - “real code” • Coupler component • Data transformations between Gridded Components • State - Data sent between Components • AppDriver – Generic driver

13. ESMF Components • “Your code goes here” • Framework specifies interface, user supplies code which is to be called • Initialize/Run/Finalize subroutine paradigm • Can create nested subcomponents

14. ESMF Gridded Components • Does the computational work • Depending on how the current code is structured, may be possible to wrap without structural changes • Or might use ESMF conversion for excuse to make structural changes!

15. ESMF Gridded Components (cont) • States for import/export • Sequential mode (concurrent coming soon) • All import data available at start • All export data ready at end • Registration routine (SetServices) to tell the framework what routines you’re supplying

16. ESMF Coupler Components • States for import/export • Does the transformation needed to take data from one Component and make it fit the requirements of another Component • Not automatic - needs to be customized for each new configuration • Expected to be thin, however - making use of the transformation routines in ESMF

17. ESMF States • Description/reference to other ESMF data objects • All data passed between Components is in the form of States and States only • Data is referenced so does not need to be duplicated • Can be Bundles, Fields, Arrays, States, or name-placeholders

18. ESMF Infrastructure Data Classes • Grids – Coordinates and grid decomposition • Arrays – Hold data • Fields – include Grid, Array, and metadata • Bundles – Fields packed together for convenience, data locality, latency reduction

19. ESMF DataMap Classes These classes give the user a systematic way of expressing interleaving and memory layout, also hierarchically (partially implemented) • ArrayDataMap – relation of array to decomposition and grid, row / column major order, complex type interleave • FieldDataMap – interleave of vector components • BundleDataMap – interleave of Fields in a Bundle

20. ESMF TimeMgr • Clocks - timesteps, time intervals • Calendars - many different varieties • Alarms - periodic, fixed time • Can be used independent of other parts of ESMF

21. ESMF Parallelism • VM - hardware and OS abstraction • DELayout - data decomposition • Grids (ESMF_GridDistribute)

22. Virtual Machine (VM) • VM handles resource allocation • Elements are Persistent Execution Threads or PETs • PETs reflect the physical computer, and are one-to-one with Posix threads or MPI processes • Parent Components assign PETs to child Components • The VM communications layer does simpleMPI-like communications between PETs (alternative communication mechanisms are layered underneath)

23. DELayout • Handles decomposition • Elements are Decomposition Elements, or DEs (decomposition that’s 2 pieces in x by 4 pieces in y is a 2 by 4 DELayout) • DELayout maps DEs to PETs, can have more than one DE per PET (for cache blocking, user-managed OpenMP threading) • Simple connectivity or more complex connectivity, with weights between DEs - users specify dimensions where greater connection speed is needed • Array, Field, and Bundle methods perform inter-DE communications

24. ESMF Communications • Halo • Updates edge data for consistency between partitions • Redistribution • No interpolation, only changes how the data is decomposed • Regrid • Based on SCRIP • Methods include bilinear, conservative, higher-order conservative • Bundle, Field, Array-level interfaces

25. ESMF Quickstart • Directory with the shell of an application • 2 Gridded Components • 1 Coupler Component • 1 Top level Gridded Component • 1 AppDriver main program

26. ESMF Quickstart Application AppDriver (generic main program) Top level Gridded Component GridComp 1 CplComp GridComp 2

27. Outline • What is ESMF? • ESMF users • A closer look at ESMF • Adoption strategies • A couple suggestions for how to get started

28. Adoption Strategies: Top Down • Wrap existing application as ESMF Component • Wrap existing data with Fields and put them into States • Can run as a Component at this point • Integrate Field level communication • Use Time Manager, Config, LogErr

29. Adoption Strategies: Bottom Up • Use ESMF utilities as needed • Incorporate the Time Manager, Config • Use Grids, Arrays for internal data • Use VM, DELayout routines for data decomposition • Wrap data structures as Fields • Use data communication routines like Halo, Regrid

30. How can you learn more? • Web site • Examples • Documentation • Interoperability experiments • Users forum

31. What do you need to get started? • Supported build platforms include SGI, IBM, Compaq, many Linuxes • C++ (gcc ok) and Fortran 90/95 compiler • MPI (or mpiuni substitute lib) • Understand how to run a job

32. Where do you go from here? • Download ESMF 2.0 and compile it • or find someone who already has it on your platform of choice! • Run the demo • Run the quickstart application • Look at the examples • Start writing code!