1 / 30

ESMF Status

ESMF Status. Climate. NCAR/LANL CCSM. Data Assimilation. GFDL FMS Suite. NASA GMAO Analysis. Weather. GMAO Seasonal Forecast. MITgcm. Cecelia DeLuca Interagency Working Group Meeting November 30, 2004. NCEP Forecast. Outline. What is ESMF? ESMF Open Source Development

radams
Download Presentation

ESMF Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ESMF Status Climate NCAR/LANL CCSM Data Assimilation GFDL FMS Suite NASA GMAO Analysis Weather GMAO Seasonal Forecast MITgcm Cecelia DeLuca Interagency Working Group Meeting November 30, 2004 NCEP Forecast

  2. Outline • What is ESMF? • ESMF Open Source Development • Development Status • Portability and Performance • Funding Status • Questions/Discussion

  3. What is ESMF? • ESMF provides tools for turning model codes into components with standard interfaces and standard drivers • ESMF provides data structures and common utilities that components use • to organize codes • to improve performance portability • for common services such as data communications, regridding, time management and message logging

  4. Why Use ESMF? • Interoperability – ESMF enables modeling groups to exchange codes, compare codes, extend codes, and use codes in different contexts • Reuse – ESMF prevents redundant development of common utilities and offers robust toolkits with many features and good documentation • Ease of Use – ESMF standard component interfaces enable modelers to understand new codes quickly, and consistent look and feel across the entire framework helps users learn new functions quickly • Performance Portability – the ESMF virtual machine provides excellent portability, good performance, and a consistent interface to diverse architectures

  5. Application Example: GEOS-5 • Each box is an ESMF component • Every component has a standard interface so that it is swappable • New components can easily be added to the system • Data in and out of components are packaged as state types • Coupling interfaces are standardized but coupling methods are customizable • Coupling tools include regridding and redistribution methods

  6. Outline • What is ESMF? • ESMF Open Source Development • Development Status • Portability and Performance • Funding Status • Questions/Discussion

  7. ESMF is a Community Effort • The ESMF collaboration has roots in the grass-roots Common Modeling Infrastructure Working Group, and grew out of Earth system modelers’ desire for shared software • Users define development priorities • Users actively test and evaluate the framework design and implementation • ~15% of ESMF source code is from user contributions (IO from WRF, resource file manager from GMAO, regridding from Los Alamos)

  8. Open Source Development • Open source license (GPL) • Open repositories: web-browsable CVS repositories for source code and contributions • Open development priorities and schedule – priorities set based on user meetings, telecons, and mailing list discussions, web-browsable task lists • Open testing: 1000+ tests are bundled with the ESMF distribution and can be run by users • Open port status: results of nightly tests on many platforms are web-browsable • Open metrics: test coverage, lines of code, requirements status are updated regularly and web-browsable

  9. Open Source Constraints • ESMF does not allow unmoderated check-ins to its main CVS repository (though there is minimal check-in oversight for the contributions repository) • ESMF has a co-located, line managed Core Team whose members are dedicated to framework implementation and support – it does not rely on a volunteer labor • ESMF actively sets priorities based on user needs and feedback • ESMF requires that contributions follow project conventions and standards for code and documentation • ESMF schedules regular releases and meetings The above are necessary for development to proceed at the pace desired by sponsors and users, and to provide the level of quality and customer support necessary for codes in this domain

  10. Outline • What is ESMF? • ESMF Open Source Development • Development Status • Portability and Performance • Funding Status • Questions/Discussion

  11. ESMF Status • Overall architecture well-defined and well-accepted • Components, States, Fields, low-level communications mature • Logically rectangular grids with regular and arbitrary distributions implemented • On-line parallel regridding (bilinear, 1st order conservative) completed and optimized • Other parallel methods, e.g. halo, redistribution, low-level comms implemented • Utilities such as time manager, logging, and configuration manager usable and adding features • Virtual machine with uniform interface to shared / distributed memory implemented, hooks for load balancing implemented

  12. ESMF Near-Term Priorities(through June 2004) • Concurrent components • Wholly irregular grid distributions • Grid merges • Optimized regridding and low-level communications, including non-blocking communications • More grids (including unstructured, tripolar, displaced pole, spectral, cubed sphere) and more regridding methods (bicubic, 2nd order conservative) • Fractional times and real times • Multi-file and other options for logging • Reworked design and implementation of array / grid / field interface and array-level communications • Asynchronous IO and read/write grid specifications

  13. ESMF Long-Term Priorities • Improve portability, performance, and error handling, and expand and improve documentation, tutorial materials, and training program • Develop and assimilate contributions of new functionality into the ESMF software (nested and adaptive grids, data assimilation support including adjoints, MPMD, improved IO) • Modify the collaboration environment and project organization so that it is effective with multiple sponsors and a larger number of collaborators • Expand the program of collaboration with CCA, PRISM and other national and international infrastructure initiatives; • Begin design and implementation of Earth System Modeling Environment (ESME)

  14. Some Metrics … • Core Team currently has 2 testers, .5 performance analyst, 5 developers, 1 admin/web support, 1 manager • 273 ESMF interfaces implemented, 250 fully or partially tested, 91% fully or partially tested. • 142,000 SLOC, 26,000 lines of text • 63 open bugs, ~316 closed bugs • 785 downloads

  15. ESMF Current Challenges • Quality and correctness of source code, especially numerical methods – recent addition of tester contributed by NASA Goddard • Process for design and interface review • Lead for development of advanced grids • Requirements database and requirements tracking – new software packages being explored • Clarity and completeness of documentation and training program materials – in short term, possible addition of casual staff for technical editing

  16. NASA Deliverable Schedule and Metrics • Public delivery of prototype ESMF v1.0 in May 2003 • Completion of first coupling demonstrations using ESMF in March 2004 • Delivered ESMF v2.0 in June 2004 • Delivery of ESMF v2.1.0 in December 2004 (includes concurrency) • All project codes scheduled to achieve partial adoption (use of the ESMF component layer and coupling) by November 2004 • All project codes scheduled to achieve full adoption (use of the component layer and coupling plus 3 or more utilities) by June 2005

  17. Outline • What is ESMF? • ESMF Open Source Development • Development Status • Portability and Performance • Funding Status • Questions/Discussion

  18. ESMF Platform Support • SGI • Altix • IBM • Compaq • Linux (Intel, PGI, NAG, Absoft, Lahey) • Mac xlf • X1

  19. ESMF Component Overhead • Measures overhead of ESMF superstructure in NCEP Spectral Statistical Analysis (SSI), ~1% overall • Run on NCAR IBM • Runs done by JPL staff, confirmed by NCEP developers

  20. ESMF Regridding Performance, Initialization • Comparison with the Argonne Model Coupling Toolkit (MCT) bundled with CCSM • Run on NCAR IBM • Runs done by JPL staff, not yet confirmed by Argonne developers

  21. ESMF Regridding Performance, Run Time • Comparison with the Argonne Model Coupling Toolkit (MCT) bundled with CCSM • Run on NCAR IBM • Runs done by JPL staff, not yet confirmed by Argonne developers

  22. Outline • What is ESMF? • ESMF Open Source Development • Development Status • Portability and Performance • Funding Status • Questions/Discussion

  23. Shared Overhead Cost *Travel, meeting, and other expenses related to project administration and management; computing time.

  24. NASA * Current development indicates that for about six external staff trying to adopt or contribute to ESMF it is useful to have 1 FTE dedicated to development/support/integration (over and above general support) on the core team.

  25. NSF

  26. NOAA

  27. DoD

  28. DoE

  29. Resulting Allocation

  30. Outline • What is ESMF? • ESMF Open Source Development • Development Status • Portability and Performance • Funding Status • Questions/Discussion

More Related