Cecelia DeLuca NOAA / CIRES University of Colorado, Boulder cecelia.deluca@noaa

1. Community infrastructure for building and coupling high performance climate, weather, and coastal models Cecelia DeLuca NOAA / CIRES University of Colorado, Boulder cecelia.deluca@noaa.gov ESMF website: www.earthsystemmodeling.org Building Community Codes for Effective Scientific Research on HPC Platforms September 6-7, 2012

2. Outline • Vision and Context • Technical Overview • User Community • Values, Processes and Governance • Future

3. Vision • Earth system models that can be built, assembled and reconfigured easily, using shared toolkits and standard interfaces. • A growing pool of Earth system modeling components that, through their broad distribution and ability to interoperate, promotes the rapid transfer of knowledge. • Earth system modelers who are able to work more productively, focusing on science rather than technical details. • An Earth system modeling community with cost-effective, shared infrastructure development and many new opportunities for scientific collaboration. • Accelerated scientific discovery and improved predictive capability through the social and technical influence of ESMF.

4. Computational Context • Teams of specialists, often at different sites, contribute scientific or computational components to an overall modeling system • Components may be at multiple levels: individual physical processes (e.g. atmospheric chemistry), physical realms (e.g. atmosphere, ocean), and members of same or multi-model ensembles (e.g. “MIP” experiments) • Components contributed from multiple teams must be coupled together, often requiring transformations of data in the process(e.g. grid remapping and interpolation, merging, redistribution) • Transformations are most frequently 2D data, but 3D is becoming more common • There is an increasing need for cross-disciplinary and inter-framework coupling for climate impacts • Running on tens of thousands of processors is fairly routine; utilizing hundreds of thousands of processors or GPUs is less common • Modelers will tolerate virtually no framework overheadand seek fault tolerance and bit reproducibility • Provenance collection is increasingly important for climate simulations

5. Architecture • The Earth System Modeling Framework (ESMF) provides a component architecture or superstructure for assembling geophysical components into applications. • ESMF provides an infrastructure that modelers use to • Generate and apply interpolation weights • Handle metadata, time management, I/O and communications, and other common functions • The ESMF distribution does not include scientific models Components Layer Gridded Components Coupler Components ESMF Superstructure User Code Model Layer ESMF Infrastructure Fields and Grids Layer Low Level Utilities External Libraries MPI, NetCDF, …

6. Components • ESMF is based on the idea of components – sections of code that are wrapped in standard interfaces • Components can be arranged hierarchically, helping to organize the structure of complex models • Different modeling groups may create different kinds or levels of components Some of the ESMF components in the GEOS-5 atmospheric GCM

7. ESMF as anInformation Layer • Parallel generation and application of interpolation weights • Run-time compliance checking of metadata and time behavior • Fast parallel I/O • Redistribution and other parallel communications • Automated documentation of models and simulations • Ability to run components in workflows and as web services Applications of information layer Structured model information stored in ESMF wrappers Attributes: CF conventions, ISO standards, METAFOR Common Information Model Standard metadata ESMF data structures Standard data structures Component Field Grid Clock User data is referenced or copied into ESMF structures Native model data structures modules grids timekeeping fields

8. Standard Interfaces All ESMF components have the same three standard methods: Initialize Run Finalize Each standard method has the same simple interface: • Steps to adopting ESMF • Divide the application into components (without ESMF) • Copy or reference component input and output data into ESMF data structures • Register components with ESMF • Set up ESMF couplers for data exchange call ESMF_GridCompRun (myComp, importState, exportState, clock, …) Where: myComp points to the component importState is a structure containing input fields exportState is a structure containing output fields clock contains timestepping information • Interfaces are wrappers and can often be set up in a non-intrusive way

9. Data Structures • ESMF represents data as Fields that can be built on four discretization types: • Logically rectangular grids, which may be connected at the edges • Unstructured meshes • Observational data streams • Exchange grids – grids that are the union of the grids of components being coupled, and reference the data on the original grids • ESMF can transform data among these representations, using an underlying finite element unstructured mesh engine

10. Component Overhead • Representation of the overhead for ESMF wrapped native CCSM4 component • For this example, ESMF wrapping required NO code changes to scientific modules • No significant performance overhead (< 3% is typical) • Few code changes for codes that are modular • Platform: IBM Power 575, bluefire, at NCAR • Model: Community Climate System Model (CCSM) • Versions: CCSM_4_0_0_beta42 and ESMF_5_0_0_beta_snapshot_01 • Resolution: 1.25 degree x 0.9 degree global grid with 17 vertical levels for both the atmospheric and land model, i.e. 288x192x17 grid. The data resolution for the ocean model is 320x384x60.

11. ESMF Regridding IMPACT: “use of the parallel ESMF offline regridding capability has reduced the time it takes to create CLM surface datasets from hours to minutes” - Mariana Vertenstein, NCAR ESMF offers extremely fast parallel regridding with many options (regional, global; bilinear, higher order, first order conservative methods; logically rectangular grids and unstructured meshes; pole options; 2D or 3D; invocation as offline application or during model run) Summary of features: http://www.earthsystemmodeling.org/esmf_releases/non_public/ESMF_5_3_0/esmf_5_3_0_regridding_status.htm Regional Grid HOMME Cubed Sphere Grid with Pentagons Courtesy Mark Taylor of Sandia FIM Unstructured Grid ESMF supported grids

12. Regridding Modes • ESMF Offline RegridWeightGen Application: • Separate application that is built as part of ESMF, can be used independently • Application generates a netCDF weight file from two netCDF grid files • Supports SCRIP, GRIDSPEC, UGRID, and custom ESMF unstructured format mpirun –np 32 ESMF_RegridWeightGen –s src_grid.nc –d dst_grid.nc –m bilinear –w weights.nc • Regridding during model execution: • ESMF library subroutine calls which do interpolation during model run • Can get weights or feed directly into ESMF parallel sparse matrix multiply • Can be used without ESMF components call ESMF_FieldRegridStore(srcField=src, dstField=dst, regridMethod=ESMF_REGRID_METHOD_BILINEAR, routehandle=rh) call ESMF_FieldRegrid(srcField=src, dstField=dst, routehandle=rh)

13. ESMP – Python Interface Flexible way to use ESMF parallel regridding Separate download: http://esmfcontrib.cvs.sourceforge.net/viewvc/esmfcontrib/python/ESMP/ Requirements: Python Numpy Ctypes Limited platform support: Linux/Darwin, GCC(g++/gfortran), OpenMPI Data type: ESMP_Field Grid types: Single-tile 2D logically rectangular type: ESMP_Grid Unstructured type: ESMP_Mesh Support for all ESMF interpolation options

14. ESMF Web Services for Climate Impacts • GOAL: Develop a two-way coupled, distributed, service-based modeling system comprised of an atmospheric climate model and a hydrological model, utilizing standard component interfaces from each domain. • IDEA: Bring the climate model to local applications • Two way technical coupling completed during 2012 using CAM (with active land) and SWAT (Soil Water Assessment Tool) • Utilized switch in CAM’s ESMF component wrapper that enables web service interface • Coupled system is run using the OpenMI configuration editor, a web service driver developed in the hydrological community • Next steps: using WRF within the CESM framework and an updated hydrological model

15. Summary of Features • Components with multiple coupling and execution modes for flexibility, including a web service execution mode • Fast parallel remapping with many features • Core methods are scalable to tens of thousands of processors • Supports hybrid (threaded/distributed) programming for optimal performance on many computer architectures; works with codes that use OpenMP and OpenACC • Time management utility with many calendars, forward/reverse time operations, alarms, and other features • Metadata utility that enables comprehensive, standard metadata to be written out in standard formats • Runs on 30+ platform/compiler combinations, exhaustive nightly regression test suite (4500+ tests) and documentation • Couples Fortran or C-based model components • Open source license

16. Major Users ESMF Components: • NOAA National Weather Service operational weather models (GFS, Global Ensemble, NEMS) • NASA atmospheric general circulation model GEOS-5 • Navy and related atmospheric, ocean and coastal research and operational models – COAMPS, NOGAPS, HYCOM, WaveWatch, others • Hydrological modelers at Delft hydraulics, space weather modelers at NCAR and NOAA ESMF Regridding and Python Libraries • NCAR/DOE Community Earth System Model (CESM) • Analysis and visualization packages: NCAR Command Language, Ultrascale Visualization - Climate Data Analysis Tools (UV-CDAT), PyFerret users • Community Surface Dynamics Modeling System

17. Usage Metrics • Updated ESMF component listing at:http://www.earthsystemmodeling.org/components/ • Includes 85 components with ESMF interfaces, 12 coupled cross-agency modeling systems in space weather, climate, weather, hydrology, and coastal prediction, for operational and research use • About 4500 registered downloads

18. Values and Principles • Community driven development and community ownership • Openness of project processes, management, code and information • Correctness • Commitment to a globally distributed and diverse development and customer base • Simplicity • Efficiency • User engagement • Environmental stewardship Web link for detail: http://www.esmf.ucar.edu/about_us/values.shtml

19. Agile Development vsCommunity Values In general, Agile processes promote to community development. However, there are areas of potential conflict: Agile emphasis on co-location: “most agile methodologies or approaches assume that the team is located in a single team room” (from Miller, Dist. Agile Development at Microsoft, 2008) vs ESMF emphasis on distributed co-development Agile emphasis on single product owner vs ESMF emphasis on multi-party ownership

20. Making DistributedCo-Development Work Hinges on asynchronous, all-to-all communication patterns: everybody must have information • Archived email list where all development correspondence gets cc’d • Minutes for all telecons • Web browsable repositories (main and contributions), mail summary on check-ins • Daily, publicly archived test results • Monthly archived metrics • Public archived trackers (bugs, feature requests, support requests, etc.) Discouraged: IMing, one-to-one correspondence or calls – the medium matters

21. Change Review Board • CRB established as a vehicle for shared ownership through user task prioritization and release content decisions • Consists of technical leads from key user communities • Not led by the development team! • Sets the schedule and expectations for future functionality enhancements in ESMF internal and public distributions • Based on broad user community and stakeholder input • Constrained by available developer resources • Updated quarterly to reflect current realities • CRB reviews releases after delivery for adherence to release plan

22. Governance Highlights • Management of ESMF required governance that recognized social and cultural factors as well as technical factors • Main practical objectives of governance: • Enabling stakeholders to fight and criticize in a civilized, contained, constructive way • Enabling people to make priority decisions based on resource realities • Observations: • Sometimes just getting everyone equally dissatisfied and ready to move on is a victory • Thorough, informed criticism is the most useful input a project can get • Governance changes and evolves over the life span of a project

23. Governance Functions • Prioritize development tasks in a manner acceptable to major stakeholders and the broader community, and define development schedules based on realistic assessments of resource constraints (CRB) • Deliver a product that meets the needs of critical applications, including adequate and correct functionality, satisfactory performance and memory use, ... (Core) • Support users via prompt responses to questions, training classes, minimal code changes for adoption, thorough documentation, ... (Core) • Encouragecommunity participation in design and implementation decisions frequently throughout the development cycle (JST) • Leverage contributions of software from the community when possible (JST) • Create frank and constructive mechanisms for feedback (Adv. Board) • Enable stakeholders to modify the organizational structure as required (Exec. Board) • Coordinate and communicate at many levels in order to create a knowledgeable and supportive network that includes developers, technical management, institutional management, and program management (IAWG and other bodies)

24. Governance ExecutiveManagement Executive Board Strategic Direction Organizational Changes Board Appointments annually Reporting Interagency Working Group Stakeholder Liaison Programmatic Assessment & Feedback Advisory Board External Projects Coordination General Guidance & Evaluation Reporting Working Project Change Review Board Development Priorities Release Review & Approval Joint Specification Team Requirements Definition Design and Code Reviews External Code Contributions quarterly Functionality Change Requests weekly Resource Constraints Implementation Schedule Collaborative Design Beta Testing Core Development Team Project Management Software Development Testing & Maintenance Distribution & User Support daily

25. Evolution Phase 1: 2002-2005 NASA’s Earth Science Technology Office ran a solicitation to develop an Earth System Modeling Framework (ESMF). A multi-agency collaboration (NASA/NSF/DOE/NOAA) won the award. The core development team was located at NCAR. A prototype ESMF software package (version 2r) demonstrated feasibility. Phase 2: 2005-2010 New sponsors included Department of Defense and NOAA. A multi-agency governance plan including the CRB was created: http://www.earthsystemmodeling.org/management/paper_1004_projectplan.pdf Many new applications and requirements were brought into the project, motivating a complete redesign of framework data structures (version 3r). Phase 3: 2010-2015 (and beyond) The core development team moved to NOAA/CIRES for closer alignment with federal models. Basic framework development completed with version 5r (ports, bugs, feature requests, user support etc. still require resources). The focus is on increasing adoption and creating a community of interoperable codes.

26. Technical Evolution:National Unified Operational Prediction Capability (NUOPC) • ESMF allows for many levels of components, types of components, and types of connections • In order to achieve greater interoperability, usage and content conventions and component templates are needed • A tri-agency collaboration (NOAA, Navy, Air Force) is building a “NUOPC Layer” that constrains how ESMF is used, and introduces metadata and other content standards, along with inheritable templates for different usage scenarios • A production version of the NUOPC Layer is scheduled for delivery at the end of 2012

27. CoG Collaboration Environment Central navigation: Templated content Right side: Services Left side: Auto generated navigation of project freeform content Central Section: Freeform content The CoG environment exposes and collates the information needed for distributed, multi-project development, including project repositories, trackers, and governance processes. It does this in an environment that’s linked to data search, metadata, and visualization services and set up to enable component comparisons.

28. DCMIP on CoG Atmospheric Dynamical Core Model Intercomparison Project http://www.earthsystemcog.org/projects/dcmip-2012/ Planned MIPs: 2013 Downscaling 2014 Atm-surface hydrology 2014 Frameworks

29. Future Governance and process largely follow established patterns Active development Python API redesign and addition of features Preparation for NUOPC Layer major release Web services for climate impacts Regridding options – higher order conservative , filling in gaps Advanced fault-tolerance Performance optimizations and GPUs CoG collaboration environment ESMF and related development is supported by NASA, NOAA, NSF, and theDepartment of Defense.