Architecture of the earth system modeling framework
1 / 37

Architecture of the Earth System Modeling Framework - PowerPoint PPT Presentation

  • Uploaded on

Architecture of the Earth System Modeling Framework. NCAR/LANL CCSM. Climate. GFDL FMS Suite. Data Assimilation. NASA GMAO Analysis. GMAO Seasonal Forecast. MITgcm. Weather. Cecelia DeLuca GEM Snowmass, CO. NCEP Forecast. Outline. Background and Motivation Applications

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Architecture of the Earth System Modeling Framework' - morag

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Architecture of the earth system modeling framework
Architecture of the Earth System Modeling Framework




Data Assimilation

NASA GMAO Analysis

GMAO Seasonal Forecast



Cecelia DeLuca


Snowmass, CO

NCEP Forecast


  • Background and Motivation

  • Applications

  • Architecture

  • Implementation

  • Status

  • Future Plans

  • Conclusions

Motivation for esmf
Motivation for ESMF

In climate research and NWP...increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system

In computing technology... increase in hardware and software complexity in high-performance computing, shift toward the use of scalable computing architectures

In software …development of frameworks, such as the GFDL Flexible Modeling System (FMS) and Goddard Earth Modeling System (GEMS) that encourage software reuse and interoperability

The ESMF is a focused community effort to tame the complexity of models and

the computing environment. It leverages, unifies and extends existing software

frameworks, creating new opportunities for scientific contribution and collaboration.

Esmf project description
ESMF Project Description

GOALS: To increase software reuse, interoperability, ease of use and performance portability in climate, weather, and data assimilation applications


  • Core framework: Software for coupling geophysical components and utilities for building components

  • Applications: Deployment of the ESMF in 15 of the nation’s leading climate and weather models, assembly of 8 new science-motivated applications


    RESOURCES and TIMELINE: $9.8M over 3 years, starting February 2002


  • Background and Motivation

  • Applications

  • Architecture

  • Implementation

  • Status

  • Future Plans

  • Conclusions

Interoperability experiments completed
Interoperability Experiments Completed


GFDL B-grid atmosphere coupled to MITgcm ocean

Atmosphere, ocean, and coupler are set up as ESMF components

Uses ESMF regridding tools


NCAR Community Atmospheric Model (CAM) coupled to MITgcm ocean

Atmosphere, ocean, and coupler are set up as ESMF components

Uses ESMF regridding tools


Temperature SSI import

Temperature SSI export

Temperature difference

NCAR Community Atmospheric Model (CAM) coupled to NCEP Spectral Statistical Interpolation (SSI) System, both set up as ESMF components

Experiment utilizes same observational stream used operationally at NCEP


  • Background and Motivation

  • Applications

  • Architecture

  • Implementation

  • Status

  • Future Plans

  • Conclusions

Characteristics of weather and climate simulation
Characteristics of Weather and Climate Simulation


  • Mix of global transforms and local communications

  • Load balancing for diurnal cycle, event (e.g. storm) tracking

  • Applications typically require 10s of GFLOPS, 100s of PEs – but can go to 10s of TFLOPS, 1000s of PEs

  • Required Unix/Linux platforms span laptop to Earth Simulator

  • Multi-component applications: componenthierarchies, ensembles, and exchanges

  • Data and grid transformations between components

  • Applications may be MPMD/SPMD, concurrent/sequential, combinations

  • Parallelization via MPI, OpenMP, shmem, combinations

  • Large applications (typically 100,000+ lines of source code)

Seasonal Forecast




sea ice







Esmf architecture
ESMF Architecture

  • ESMF provides an environment for assembling geophysical components into applications, with support for ensembles and hierarchies.

  • ESMF provides a toolkit that components use to

    • increase interoperability

    • improve performance portability

    • abstract common services

Hierarchies and ensembles
Hierarchies and Ensembles

ESMF encourages applications to be assembled hierarchically and intuitively

Coupling interfaces are standard at each layer

Components can be used in different contexts

ESMF supports ensembles with multiple instances of components running sequentially (and soon, concurrently)

Ensemble Forecast

Seasonal Forecast


sea ice












Esmf class structure
ESMF Class Structure


Land, ocean, atm, … model


Xfers between GridComps


Data imported or exported




Collection of fields


Computes interp weights


Physical field, e.g. pressure


LogRect, Unstruct, etc.


Math description


Grid decomposition



Hybrid F90/C++ arrays




Stores comm paths



Machine, TimeMgr, LogErr, I/O, Config, Base etc.



Esmf data classes
ESMF Data Classes

Model data is contained in a hierarchy of multi-use classes. The user can reference a Fortran array to an Array or Field, or retrieve a Fortran array out of an Array or Field.

  • Array – holds a cross-language Fortran / C++ array

  • Field – holds an Array, an associated Grid, and metadata

  • Bundle – collection of Fields on the same Grid

  • State – contains States, Bundles, Fields, and/or Arrays

  • Component – associated with an Import and Export State

Esmf datamap classes
ESMF DataMap Classes

These classes give the user a systematic way of expressing interleaving and memory layout, also hierarchically (partially implemented)

  • ArrayDataMap – relation of array to decomposition and grid, row / column major order, complex type interleave

  • FieldDataMap – interleave of vector components

  • BundleDataMap – interleave of Fields in a Bundle

Esmf standard methods
ESMF Standard Methods

ESMF uses consistent names and behavior throughout the framework, for example

  • Create / Destroy – create a new object, e.g. FieldCreate

  • Set / Get – set or get a value, e.g. ArrayGetDataPtr

  • Add / Get / Remove – add to, retrieve from, or remove from a list, e.g. StateAddField

  • Print – to print debugging info, e.g. BundlePrint

  • And so on


  • ESMF is a standard component architecture, similar to CCA but designed for the Earth modeling domain and for ease of use with Fortran codes

  • Components and States are superstructure classes

  • All couplers are the same derived type (ESMF_CplComp) and have a standard set of methods with prescribed interfaces

  • All component models (atm, ocean, etc.) are the same derived type (ESMF_GridComp) and have a standard set of methods with prescribed interfaces

  • Data is transferred between components using States.

  • ESMF components can interoperate with CCA components – demonstrated at SC03

Becoming an esmf gridcomp
Becoming an ESMF GridComp

  • ESMF GridComps have 2 parts: one part user code, one part ESMF code

  • The ESMF part is a GridComp derived type with standard methods including Initialize, Run, Finalize

  • User code must also be divided into Initialize, Run, and Finalize methods – these can be multi-phase (e.g. Run phase 1, Run phase 2)

  • User code interfaces must follow a standard form – that means copying or referencing data to ESMF State structures

  • Users write a public SetServices method that contains ESMF SetEntryPoint calls - these associate a user method (“POPinit”) with a framework method (the Initialize call for a GridComp named “POP”)

  • Now you’re an ESMF GridComp


  • Data classes are Bundles, Fields, and Arrays

  • Tools for expressing interleaved data stuctures

  • Tools for resource allocation, decomposition, load balancing

  • Toolkits for communications, time management, logging, IO

Virtual machine vm
Virtual Machine (VM)

  • VM handles resource allocation

  • Elements are Persistent Execution Threads or PETs

  • PETs reflect the physical computer, and are one-to-one with Posix threads or MPI processes

  • Parent Components assign PETs to child Components

  • PETs will soon have option for computational and latency / bandwidth weights

  • The VM communications layer does simpleMPI-like communications between PETs (alternative communication mechanisms are layered underneath)


  • Handles decomposition

  • Elements are Decomposition Elements, or DEs (decomposition that’s 2 pieces in x by 4 pieces in y is a 2 by 4 DELayout)

  • DELayout maps DEs to PETs, can have more than one DE per PET (for cache blocking, user-managed OpenMP threading)

  • A DELayout can have a simple connectivity or more complex connectivity, with weights between DEs - users specify dimensions where greater connection speed is needed

  • DEs will also have computation weights

  • Array, Field, and Bundle methods perform inter-DE communications

Esmf communications
ESMF Communications

  • Communication methods include Regrid, Redist, Halo, Gather, Scatter, etc.

  • Communications methods are implemented at multiple levels, e.g. FieldHalo, ArrayHalo

  • Communications hide underlying ability to switch between shared and distributed memory parallelism

Load balancing
Load Balancing

Three levels of graphs:

  • Virtual Machine – machine-level PETs will have computational and connectivity weights

  • DELayout – DE chunks have connectivity weights, will have computational weights

  • Grid – grid cells will have computational and connectivity weights

    Intended to support standard load balancing packages (e.g. Parmetis) and user-developed load balancing schemes


  • Background and Motivation

  • Applications

  • Architecture

  • Implementation

  • Status

  • Future Plans

  • Conclusions

Open development
Open Development

  • Open source

  • Currently ~800 unit tests, ~15 system tests are bundled with the ESMF distribution, can be run in non-exhaustive or exhaustive modes

  • Results of nightly tests on many platforms are accessible on a Test and Validation webpage

  • Test coverage, lines of code, requirements status are available on a Metrics webpage

  • Exhaustive Reference Manual, including design and implementation notes, is available on a Downloads and Documentation webpage

  • Development is designed to allow users clear visibility into the workings and status of the system, to allow users to perform their own diagnostics, and to encourage community ownership

Port status
Port Status

  • SGI

  • IBM

  • Compaq

  • Linux (Intel, PGI, NAG, Absoft, Lahey)


  • Background and Motivation

  • Applications

  • Architecture

  • Implementation

  • Status

  • Future Plans

  • Conclusions

Esmf key accomplishments
ESMF Key Accomplishments

  • Public delivery of prototype ESMF v1.0 in May 2003

  • Monthly ESMF internal releases with steadily increasing functionality

  • Completion of first 3 coupling demonstrations using ESMF in March 2004

    • NCAR CAM with NCEP SSI

    • NCAR CAM with MITgcm ocean

    • GFDL B-grid atmosphere with MITgcm ocean

    • All codes above running as ESMF components and coupled using the framework, codes available from Applications link on website

    • Other codes running as ESMF components: MOM4, GEOS-5

    • Less than 2% lines of source code change

  • Delivered ESMF v2.0 in June 2004

  • 3rd Community Meeting to be held on 15 July 2004 at NCAR

Esmf priorities
ESMF Priorities


  • Components, States, Bundles, Fields mature

  • On-line parallel regridding (bilinear, 1st order conservative) completed

  • Other parallel methods, e.g. halo, redist, low-level comms implemented

  • Comm methods overloaded for r4 and r8

  • Communications layer with uniform interface to shared / distributed memory, hooks for load balancing

    Near-term priorities

  • Concurrent components – currently ESMF only runs in sequential mode

  • More optimized grids (tripolar, spectral, cubed sphere) and more regridding methods (bicubic, 2nd order conservative) from SCRIP

  • Comms optimization and load balancing capability

  • IO (based on WRF IO)

  • Development schedule on-line, see Development link, SourceForge site tasks


  • Background and Motivation

  • Applications

  • Architecture

  • Implementation

  • Status

  • Future Plans

  • Conclusions

Next steps
Next Steps

  • Integration with data archives and metadata standardization efforts, anticipate collaboration with Earth System Grid (ESG) and European infrastructure project PRISM

  • Integration with scientific model intercomparison projects (MIPs), anticipate collaboration with the Program for Climate Model Diagnosis and Intercomparison (PCMDI), other community efforts

  • Integration with visualization and diagnostic tools for end-to-end modeling support, anticipate collaboration with the Earth Science Portal (ESP)

  • ESMF “vision” for the future articulated in multi-agency white paper on the Publications and Talks webpage

Esmf multi agency follow on
ESMF Multi-Agency Follow-on

  • 3 ESMF FTEs at NCAR slated to have ongoing funding through core NCAR funds

  • NASA commitment to follow-on support, level TBD

  • DoD and NSF proposals outstanding

  • Working with other agencies to secure additional funds


  • Background and Motivation

  • Applications

  • Architecture

  • Implementation

  • Status

  • Future Plans

  • Conclusions

Esmf overall
ESMF Overall

  • Clear, simple hierarchy of data classes

  • Multi-use objects mean that the same object can carry information about decomposition, communications, IO, coupling

  • Tools for multithreading, cache blocking, and load balancing are being integrated into the architecture

  • Objects have consistent naming and behavior across the framework

The benefits
The Benefits

  • Standard interfaces to modeling components promote increased interoperability between centers, faster movement of modeling components from research to operations

  • The ability to construct models hierarchically enables developers to add new modeling components more systematically and easily, facilitates development of complex coupled systems

  • Multi-use objects mean that the same data structure can carry information about decomposition, communications, IO, coupling – this makes code smaller and simpler, and therefore less bug-prone and easier to maintain

  • Shared utilities encourage efficient code development, higher quality tools, more robust codes

More information
More Information

ESMF website:


The ESMF is sponsored by the NASA

Goddard Earth Science Technology Office.