Scidac ccsm consortium
This presentation is the property of its rightful owner.
Sponsored Links
1 / 26

SciDAC CCSM Consortium PowerPoint PPT Presentation


  • 73 Views
  • Uploaded on
  • Presentation posted in: General

SciDAC CCSM Consortium. Overview What is the purpose of the project? What is accomplished? Where are we going?.

Download Presentation

SciDAC CCSM Consortium

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Scidac ccsm consortium

SciDAC CCSM Consortium

Overview

What is the purpose of the project?

What is accomplished?

Where are we going?


Scidac ccsm consortium

The SciDAC CCSM Consortium consists of PI: R. Malone4, J. Drake5 , Site-Contacts: C. Ding2, S. Ghan6, D. Rotman3, J. Taylor1, J. Kiehl7, W. Washington7, S.-J. Lin8, Co-Is: J. Baumgardner4, T. Bettge7, L. Buja7, S. Chu4, T. Craig7, P. Duffy3, J. Dukowicz4, S. Elliot4, D. Erickson5, M. Ham5, Y. He2, F. Hoffman5, E. Hunke4, R. Jacob1, P. Jones4, J. Larson1, J. Lamarque7, W. Lipscomb4, M. Maltrud4, D. McKenna7, A. Mirin3, W. Putman8, W. Sawyer8, J. Schramm7, T. Shippert6, R. Smith4, P. Worley5, W. Yang2

1Argonne National Lab, 2Lawrence Berkeley National Lab, 3Lawrence Livermore National Lab, 4Los Alamos National Lab, 5Oak Ridge National Lab, 6Pacific Northwest National Lab, 7National Center for Atmospheric Research, 8NASA-Goddard Space Flight Center


Scidac ccsm consortium

http://www.osti.gov/scidac/updates2003.html

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Why a community model

Why a Community Model?

  • NCAR adopted to encourage climate research in universities using state of the art models. –1980

  • New components of a coupled system. - 1990

  • Individual projects unable to adapt to parallel architectures -2000

  • Pool resources and complete mission critical simulations for NSF, DOE and NASA

  • CCSM has become the “national model”

  • Annual Breckenridge CCSM Workshop grown to over 270 researchers

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Staged software engineering development cycle

Staged Software Engineering Development Cycle

  • Write and review requirements list

  • Write and review a design document

  • Write and review prototype code

  • Unit test code

  • Integrate and validate code

  • Update design document and update user documentation

    How do you do this and maintain science ready code?

  • CCSM2.0 Released May 2002

  • CCSM2.0.1 Released Oct. 2002 (bit-for-bit climate)

  • Science mods for IPCC Assessment now in the works

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Scidac ccsm consortium

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Scidac ccsm consortium

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Critical decisions for software development

Critical Decisions for Software Development

  • Software Engineering Working Group added to CCSM

  • Select DOE & NASA programmers given developer access

  • Hybrid distributed/shared memory programming paradigm

  • Transpose based parallel algorithms, cache & vector friendly

  • Maintain usable code – no down time for scientists!

  • Develop extensive testing procedures

  • Component gatekeepers or Change Review Boards

  • Target platform list: NCAR, ORNL, LANL, NERSC

  • Support CCSM priorities: IPCC development

New

New

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Doe scidac workshop on porting ccsm to the cray x1

DOE SciDAC Workshop on Porting CCSM to the CRAY X1

  • Goals

    • Identify individuals and organizations engaged in porting one or more of the CCSM component models

    • Report progress and problems in current CCSM vectorization activities.

    • Identify gaps or issues in the current efforts.

    • Establish lines of communication between the different efforts and NCAR software engineers to encourage sharing of results and code.

    • Begin defining requirements and procedures for the adoption ofvector-friendly code in future released versions of CCSM.

  • Represented: NCAR, NASA-Goddard, ORNL, LANL, LBNL, Cray, NEC, Fujitsu, CRIEPI

  • Held Feb. 2003 in Boulder, also June in Breckenridge 2003

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Coupler architecture tony craig rob jacob brian kaufman jay larson e ong

Coupler ArchitectureTony Craig, Rob Jacob, Brian Kaufman, Jay Larson, E. Ong

  • Issues:

  • sequencing

  • frequency

  • distribution

  • parallelism

  • single or multiple

  • executables

  • stand alone execution

  • MPH3 (multi-processor handshaking) library for coupling component models

  • CPL6 -- Implemented, Tested, Deployed

Version 1.0 Released

November 2002

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Evolution of performance of the community atmospheric model pat worley john drake

Evolution of Performance of the Community Atmospheric ModelPat Worley, John Drake

  • CAM2.0 Eulerian Spectral Model at T42L26

  • IBM p690

  • Hybrid MPI/OpenMP programming paradigm

  • Cache friendly chunks, load balance, improved algorithms

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Performance of the cam2 with fv core art mirin will sawyer bill putman

Performance of the CAM2 with FV CoreArt Mirin, Will Sawyer, Bill Putman

  • FV Core on IBM NHII in lat-vert decomposition

  • NASA- DAO’s mod_comm replacing PILGRIM

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Land surface model and river transport model forrest hoffman mariana verenstein marcia branstetter

Land Surface Model and River Transport Model Forrest Hoffman, Mariana Verenstein, Marcia Branstetter

Community Land Model (CLM2.1) Released

  • SciDAC software engineering is focused on the interface and reduction of gather/scatters; communications bottleneck removed

  • Rewrite for vectorization and CLM2.2 now complete

  • RTM is currently single processor -- designing parallel implementation and data structures

  • Analysis of runoff in CCSM control simulation. Effect on July ocean salinity.

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Parallel i o woo sun yang chris ding

Distributed array

In (X,Z,Y) index order

Remapped on staging PEs

In (X,Y,Z) index order

I/O staging PEs

write in parallel

height

(Z)

latitude (Y)

Z-decomposition

longitude (X)

MPI gather, global transpose and single-PE write

MPI gather and global transpose

Total write times with ZioLib

Remapping times with ZioLib

Parallel I/OWoo-Sun Yang, Chris Ding

Parallel decompositions of ZioLib

Parallel performance of Ziolib

Parallel NetCDF in conjunction with SciDAC SDM Center

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Pop ocean model p jones j dukowicz j baumgardner w lipscomb

POP Ocean ModelP. Jones, J. Dukowicz, J. Baumgardner, W. Lipscomb

  • Software Engineering for POP and CICE

    • Design and implementation for the new ocean model (HYPOP) and CICE in progress

  • Ocean Model Performance

    • POP2: new design involves a decomposition of the computational domain into blocks that can be sized to fit into cache

    • On 1/10 degree, SGI (2x), IBM (1.25x), long vector gets 50% peak on Fujitsu

  • MLP in POP

    • On 1/10 degree, SGI(2x)

  • HYPOP Model Development

    • Treat purely Lagrangian dynamics of constant-mass layers as they inflate and deflate in regions intersecting bottom topography

    • Pressure gradient is split into a 'baroclinic' part that vanishes and a 'barotropic' part that does not vanish when the density is uniform

    • Comparison of surface height in Lagrangian and Eulerian vertical after 400 baroclinic steps

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Ocean generic grid generator ecubed w voit j davis r smith

Ocean generic Grid Generator – eCubedW. Voit, J. Davis, R. Smith

  • GUI in Tcl/Tk

  • Tripole grid supported

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Sea ice model j schramm p jones w lipscomb

Sea Ice ModelJ. Schramm, P. Jones, W. Lipscomb

  • Incremental Remapping for Sea Ice and Ocean Transport

    • Incremental remapping scheme that proved to be three times faster than MPDATA, total model speedup of about 30% --added to CCSM/CSIM

    • Cache and vector optimizations

    • CICE3.0 restructered for vector Community Sea Ice Model

  • Sensitivity analysis and parameter tuning test of the CICE code

    • Automatic Differentiation (AD)-generated derivative code

    • Major modeling parameters that control the sea ice thickness computation were the ice-albedo constants, densities and emissivities of ice and snow, and salinity constant

    • Parameter tuning experiment with gradient information

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Resolution and precipitation phil duffy

Resolution and PrecipitationPhil Duffy

(DJF) precipitation in the California region in 5 simulations, plus observations. The 5 simulations are: CCM3 at T42 (300 km), CCM3 at T85 (150 km) , CCM3 at T170 (75 km), CCM3 at T239 (50 km), and CAM2 with FV dycore at 0.4 x 0.5 deg.

CCM3 extreme precipitation events depend on model resolution. Here we are using as a measure of extreme precipitation events the 99th percentile daily precipitation amount. Increasing resolution helps the CCM3 reproduce this measure of extreme daily precipitation events.

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Subgrid orography scheme steve ghan tim shippert

Subgrid Orography SchemeSteve Ghan, Tim Shippert

  • Reproduces orographic signature without increasing dynamic resolution

  • Realisitic precipitation, snowcover, runoff

  • Month of March simulated with CCSM

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Atmospheric chemistry p cameron smith j taylor d erickson j f lamarque s walters d rotman

Atmospheric ChemistryP. Cameron-Smith, J. Taylor, D. Erickson, J.F. Lamarque, S. Walters, D. Rotman

  • Gas-phase chemistry with emissions, deposition, transport and photo-chemical reactions for 89 species.

  • Experiments performed with 4x5 degree Fvcore – ozone concentration at 800hPa for selected stations (ppmv)

  • Mechanism development with IMPACT

    • A)    Small mechanism (TS4), using the ozone field it generates for photolysis rates.

    • B)     Small mechanism (TS4), using an ozone climatology for photolysis rates.

    • C)    Full mechanism (TS2), using the ozone field it generates for photolysis rates.

Zonal mean

Ozone, Ratio A/C

Zonal mean

Ozone, Ratio B/C

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Ocean biogeochemistry s elliot s chu m maltrud

Ocean BiogeochemistryS. Elliot, S. Chu, M. Maltrud

  • Iron Enrichment in the Parallel Ocean Program

  • Surface chlorophyll distributions in POP

  • for 1996 La Niña and 1997 El Niño

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Global dms flux from the ocean using pop d erickson j hernandez m maltrud s chu

Global DMS Flux from the Ocean using POPD. Erickson, J. Hernandez, M. Maltrud, S. Chu

The global flux of DMS from the ocean to the atmosphere is shown as an annual mean. The globally integrated flux of DMS from the ocean to the atmosphere is 23.8 Tg S yr-1 .

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Goals of the consortium

Goals of the Consortium

  • Performance portability for the CCSM

  • Open software design process

  • Layered software architecture to insulate modeling

  • Readiness for global to regional climate change simulations

  • High fidelity ocean and ice models

  • Extension of atmospheric chemistry capability

  • Development of biogeochemical, terrestrial and hydrological aspects of the CCSM

  • Towards comprehensive coupled climate simulations for study of decadal to century climate change.

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Simulation of future climates is a very hard problem

Simulation of Future Climatesis a “very hard problem”

Computing for these simulations was done

at DOE's National Energy Research Scientific Computing Center (NERSC), at NCAR’s Climate System Laboratory (CSL), at LANL’s Advanced Computing Laboratory (ACL) and at ORNL’s Center for Computational Sciences (CCS).

Credits

 The SciDAC CCSM Consortium consists of PI: R. Malone4, J. Drake5 , Site-Contacts: C. Ding2, S. Ghan6, D. Rotman3, J. Taylor1, J. Kiehl7, W. Washington7, S.-J. Lin8, Co-Is: J. Baumgardner4, T. Bettge7, L. Buja7, S. Chu4, T. Craig7, P. Duffy3, J. Dukowicz4, S. Elliot4, D. Erickson5, M. Ham5, Y. He2, F. Hoffman5, E. Hunke4, R. Jacob1, P. Jones4, J. Larson1, J. Lamarque7, W. Lipscomb4, M. Maltrud4, D. McKenna7, A. Mirin3, W. Putman8, W. Sawyer8, J. Schramm7, T. Shippert6, R. Smith4, P. Worley5, W. Yang2

1Argonne National Lab, 2Lawrence Berkeley National Lab, 3Lawrence Livermore National Lab, 4Los Alamos National Lab, 5Oak Ridge National Lab, 6Pacific Northwest National Lab, 7National Center for Atmospheric Research, 8NASA-Goddard Space Flight Center

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


Coordination and management

NOPP

SciDAC

LANL

SCRIP

NCAR

PRISM

CSU

POP

NPS

CICE

ORNL

UCLA

CCPP

ANL

CCSM

LLNL

ESMF

Coordination and Management

  • Managment Plan

  • Memorandum of Understanding between SciDAC and ESMF

  • Coordination with SciDAC Integrated Software Infrastructure Centers and Collaboratories

    • PERC, ESG, SDM, TSTT, CCA, APDEC, TOPS, …

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


The end

The End

Review of SciDAC CCSM Consortium Project @ Oak Ridge National Laboratory


  • Login