1 / 33

Plan

Plan. Code coupling How CERFACS came into play OASIS: Historical development and scientific use at CERFACS An example of active collaborations Community today PRISM The OASIS4 coupler PALM: Contexte, principe, utilisation L’interface graphique pre_PALM

dewei
Download Presentation

Plan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Plan • Code coupling • How CERFACS came into play • OASIS: • Historical development and scientific use at CERFACS • An example of active collaborations • Community today • PRISM • The OASIS4 coupler • PALM: • Contexte, principe, utilisation • L’interface graphique pre_PALM • L’analyse de performance de pre_PALM • Le développement, les utilisateurs • OASIS et PALM: en résumé • En conclusions

  2. What: • Manage execution of the codes • Ensure synchronised data exchange between the codes • How: • 1- Merge the codes: • Efficient, but… • Integration problem (common, units, …) • Memory problem • Low flexibility prog1 & prog2 Program prog1 … call sub_prog2(in, out) … end Program prog2 Subroutine sub_prog2(in,out) … end Prog1 Prog2 données Code coupling • Why: • Treat a global system (ocean-atmosphere, fluid-structure, etc.) • Create modular applications by coupling elementary component

  3. How: • 2- Use communication protocol: • PVM, MPI, file exchange • More or less efficient • Non generic • Non flexible • Complex to implement Prog1 Prog2 Program prog1 … if ( ) call x_send(data, tag, target) endif end Program prog2 … if ( ) call x_recv(data, tag, source) endif end • 3- Use a coupler: • Maybe less efficient than 1- • User external configuration • Generic, flexible • Portable • Interpolation by coupler • Redistribution by coupler PALM/OASIS Prog1 Prog2 Program prog1 … call palm/prism_put(data, time) end Program prog2 … call palm/prism(data, time) end Code coupling

  4. Models at relatively low resolution (~10000-20000 pts) • Small number of 2D coupling fields (~10) • Low coupling frequency (~once/day) How CERFACS came into play: OASIS • The need in 1991: • To build a French ocean-atmosphere coupled model based on existing GCMs developed independently. • Allow component model intercomparison.

  5. The answer :OASISOceanAtmosphereSeaIceSoil O A O A OASIS O A O Static coupling: all coupling parameters are fixed initially by the user in an input file namcouple Flexibility, modularity, (efficiency) How CERFACS came into play: OASIS • 1 monoprocess coupler: • synchronisationof component models • exchangeof coupling fields • interpolationof coupling fields • Arbitrary number of (parallel) models and coupling fields • Different coupling frequencies for the different fields • Particular transformations for each field

  6. OASIS: Historical development and scientific use at CERFACS • 1993: OASIS1.0 (CRAY pipes and files) 10 year tropical coupling • 1995: OASIS2.0 (PVM3) CATHODE: distributed coupling • 1996: OASIS2.1 SIDDACLICH (EU): global warming scenarios • 1998: OASIS2.2 (SIPC) DEMETER (EU): seasonal forecasting • 1999: OASIS2.3 (GMEM - NEC) SINTEX (EU) • 2000: OASIS2.4 (MPI2) PREDICATE (EU) : climate natural variability • 2004: OASIS3 (MPI1 + PSMILe + I/O) PRISM + DYNAMITE (EU) • 2005: OASIS4 (full parallelisation) PRISM + GEMS (EU)

  7. MPI-M MPI-M&D NEC-CCRLE NEC-HPCE RPN SGI SMHI UKMO CGAM CRAY ECMWF EDF FECIT/Fujitsu IDRIS IPSL Météo-France OASIS: An example of active collaborations

  8. OASIS community today • CERFACS (France) ARPEGE3 - ORCA2LIM ARPEGE3 - OPA 8.1 ARPEGE3 - OPAICE • METEO-FRANCE (France)ARPEGE4 - ORCA2 ARPEGE medias -OPAmed ARPEGE3 - OPA8.1 ARPEGE2 - OPA TDH • IPSL- LODYC, LMD, LSCE (France) LMDz - ORCA2LIM LMDz - ORCA4 LMDz - OPA ATL3/ATL1 IFS - OPA 8.1 ECHAM4 - ORCA2 • MERCATOR (France) PAM(OPA) • MPI - M&D (Germany) ECHAM5 - MPI-OM ECHAM5 - C-HOPE PUMA - C-HOPE EMAD - E-HOPE ECHAM5 - E-HOPE ECHAM4 - E-HOPE • ECMWF(UK)IFS Cy23r4 - E-HOPE IFS Cy15r8 - E-HOPE • IFS - CTM

  9. OASIS community today • CGAM-Reading (UK) HadAM3 - ORCA2 • Southampton University (UK) Inter. Atm - OCCAM lite • UCL (Belgium) LMDz - CLIO • SMHI (Sweden) ECHAM - RCA RCA(region.) – RCO(region.) • U. of Bergen (Norway) MM5 - ROMS • KNMI (Netherlands) ECHAM5 - MPI-OM • DMI (Danemark) ECHAM - HIRLAM • INGV (Italy) ECHAM5 – MPI-OM • IRI (USA) ECHAM4 - MOM3 • JAMSTEC (Japan) ECHAM4 - OPA 8.2 • BMRC (Australia) BAM - MOM4 BAM3 - ACOM2 • U. of Tasmania (Australia) Data Atm. - MOM4 • RPN-Environment Canada (Canada) MEC - GOM

  10. OASIS development today: PRISM OASIS3 and OASIS4 currently developed within PRISM (PRogram for Integrated Earth System Modelling) • PRISM is organised under ENES (European Network for Earth System modelling): • «Think tank » to organize Earth System modelling in Europe • 50 members so far; open to any institute/industry (MoU) • PRISM was a European project • 22 partners funded in 2001-2004 for 4.8 M€ • Coordinators: Guy Brasseur (MPI, Hamburg) and Gerbrand Komen (KNMI, DeBilt); Director: Reinhard Budich (MPI) • Provide a standard software infrastructurefor Earth system modelling based on state-of-art European components models

  11. PRISM continues with the PRISM Support Initiative • 12 partners • own manpower for a total of about 9 py/y • support and develop the PRISM software (2005-2008): - MPI/M&D - UK Met Office - CNRS - CERFACS - ECMWF - SMHI - CGAM - CCRLE - CRAY - NEC-HPCE - SGI - SUN OASIS development today: PRISM

  12. Need to optimise and parallelise the coupler The OASIS4 coupler • Situation today: • Higher resolution, parallel and scalable models • Higher coupling frequencies desirable • Higher number of models and (3D) coupling fields • Massively parallel platforms • OASIS4 is composed of: • a Driver • a Transformer • a new PRISM System Model Interface Library (PSMILe)

  13. OASIS4: Some key notes • Developers: CERFACS; NEC CCRLE; SGI; NEC HPCE • Public domain; open source license (LGPL) • Programming language: Fortran 90 and C • Public domain libraries; vendor versions may exist: • MPI1 and/or MPI2; NetCDF/parallel NetCDF; libXML • mpp_io; SCRIP

  14. OASIS4: the Driver • Manages a static coupling: • launches models at the beginning of the run (MPI2) or • models started directly in the run script (MPI1) • Participates to communicator set-up • Reads the XML user-defined configuration info and distributes it to the component PSMILes

  15. C C C C C C OB OB O2 O1 OB OB C C O2 O1 OB OB C C T T • Different grid and decomposition • via parallel Transformer: • Same grid, different decomposition • direct repartitioning: OASIS4: the communication (1/2) • Model interface library: PSMILe based on MPI • Parallel communication including repartitioning: • based on geographical description of the partitions • parallel calculation of communication patterns in source PSMILe

  16. file A file1 A A A file A A file2 A A A file3 single parallel distributed OASIS4: the communication (2/2) • end-point communication (prism_put, prism_get) • parallel 3D search in each source process PSMILe • parallel multigrid search algorithm: high efficiency • extraction of useful part of source field only • one-to-one, one-to-many • parallel I/O : GFDL mpp_io, parNetCDF

  17. For grids: • Regular in lon, lat, vert: lon(i), lat(j), height(k) • Irregular in lon and lat, regular in vert: lon(i,j), lat(i,j), height(k) • Irregular in lon, lat, and vert: lon(i,j,k), lat(i,j,k), height(i,j,k) • Gaussian Reduced in lon and lat, reg in vert: lon(npthor), lat(npthor), height(k) OASIS4: the field transformations • source time transformations (below prism_put): • average, accumulation • target time transformations (below prism_get): • time interpolation (for I/O only) • statistics • local transformations: • addition/multiplication by scalar • interpolation/regridding (3D): • nearest-neighbour, bilinear, bicubic, trilinear

  18. OASIS4: the users • OASIS4 was tested and run with toy models on: • Intel Pentium 4 Workstation Cluster, • SGI O3000/2000, • SGI IA64 Linux server Altix 3000, • NEC SX6, • AMD 2800 Cluster, • IBM Power 4 • OASIS4 starts being used in real applications: • GFDL: MOM4-toy atmosphere • SMHI: regional coupling • IFM-GEOMAR (Kiel): interpolation of high-resolution fields. • GEMS: operational global monitoring of atm chemistry and dynamics

  19. MERCATOR:océanographie opérationnelle (1997) Spatial observations Model Forecast In-situ observations PALM: le contexte • Implémentation modulaire de chaînes d’assimilation de données • - recherche de méthodes • flexibilité - exploitation opérationnelle • performances

  20. Couplage dynamique PALM: le principe • Les différentes techniques d’assimilation de données peuvent se décomposer en opérations élémentaires • Un enchaînement donné de ces opérations élémentaires (unités) génère une technique d’assimilation particulière • L’outil PALM assure (suivant la description faite par l’utilisateur) • l’enchaînement et la synchronisation des unités • les échanges efficaces d’objets entre les unités

  21. PALM: l’utilisation • 1. Décomposer l’application en unités de traitement: • décrire les données produites (output) ou requises (input) de l’unités :objets • implémenter dans l’unité les requêtes d’objets input :PALM_GET • implémenter dans l’unité les mises à disposition d’objets output:PALM_PUT • 2. Utiliser l’interface graphique pré-palm pour: • décrire l’enchaînement dynamique de ces unités (for, while, if , select): les branches • décrire les échanges d’objets entre les différentes unités • 3. Compiler, lancer l’application et laisser PALM assurer • l’enchaînement et la synchronisation des unités décrits avec pre-Palm • les échanges efficaces d’objets entre les unités

  22. PALM: l’interface graphique pre_PALM

  23. PALM: l’analyse de performance de pre_PALM

  24. Pre_ • Interface utilisateur • Version SPMD (basee sur MPI1) : • Version MPMD (basee sur MPI2): _research _MP >Disponibles gratuitement pour la recherche http://www.cerfacs.fr/~palm PALM: le développement

  25. PALM: les utilisateurs • Projet MERCATOR: • assimilation de données en océanographie opérationnelle • Projet Européen ASSET: • assimilation de données en chimie atmosphérique (ENVISAT) • Projet ADOMOCA (PNCA): • PALM choisi comme outil fédérateur pour l’assimilation de données en chimie atmosphérique en France • EDF: • assimilation de données dans les cœurs de réacteurs nucléaires • SNECMA: • couplage 3D fluide–radiatif–solide (simulation des températures de paroi des chambres de combustion de moteur d’avion). • elsA (CERFACS): • couplage fluide-structure • Projet OPAVAR (CERFACS): • assimilation variationnelle 3/4DVAR dans l'océan • CESBIO, SCHAPI, TURBOMECA, IUSTI, IMFT,…

  26. OASIS et PALM: en résumé • OASIS: coupleur statique, interpolation parallèle -> modélisation climatique • Utilisé par plusieurs groupes internationalement • OASIS3 -> OASIS4 (version « grand public » prévue d’ici 1 an) • PALM: coupleur dynamique -> assimilation de données + autres couplages dynamiques • Communauté grandissante d’utilisateurs • Quelques améliorations (objets dyn, IO parallèles) en développement • OASIS et PALM répondent à des besoins différents de communautés différentes mais voisines • Fusion des coupleurs à + ou – long terme? • .

  27. En conclusion • Beaucoup d’interaction avec la communauté des utilisateurs • Nombreuses collaborations • Bénéfices de l’approche « open-source » (OASIS) • Besoin d’une intégration centrale forte • Au CERFACS, la coexistence d’activités de développement de logiciels et d’actions de recherche au sein d’une même équipe est extrêmement bénéfique! • .

  28. Fin

  29. OASIS4: component model description • Application and component description (XML files): • For each application (code): one Application Description (AD): • possible number of processes, components included, etc. • For each component in the application: one Potential Model Input and Output Description (PMIOD) • component general characteristics: name, component simulated, … • grid information: domain, resolution(s), grid type, … • potential I/O or coupling variables: • local name, standard name (NetCDF CF convention) • units, valid min and max • numerical type • associated grid and points • intent –input and/or output

  30. OASIS4: coupled model configuration • Coupled model configuration (XML files): • (Through a GUI,) the user produces • a Specific Coupling Configuration (SCC): • start date and end date • start mode (MPI1, MPI2) • applications, components for each application • host(s), number of processes per host, ranks for each component • For each component, • a Specific Model Input and Output Configuration (SMIOC) • grid information: chosen resolution, … • I/O or coupling variables: • local and standard name, units, valid min max, numerical type, grid • activated intent –input and/or output • source and/or target (component and/or file) • coupling or I/O dates • transformations/interpolations

  31. ATM-LAND AD Driver ATM PMIOD V3: in, metadata V3 V4: out, metadata V4 V5: out, metadata V5 user user OCE AD user OCE PMIOD V1: out, metadata V1 V2: in, metadata V2 user • SCC • ATM:... • OCE:... • LAND:... OCE OCE SMIOC V1 : to ATM V3, T1 to file V1 V2 : from ATM V4, T2 ATM ATM SMIOC V3 : from OCE V1, T1 V4: to OCE V2, T2 V5 : to LAND V6 T LAND V4 V2 V5 V6 V3 V1 LAND PMIOD V6: in, metadata V6 V7: in, metadata V7 user V7 fileV1 fileV7 Definition Phase LAND SMIOC V6 : from ATM V5 V7 : from fileV7 Composition Phase Deployment Phase

  32. x x x x x x x x x x x x x x x x x x x x o o o o o o o o o 3/ EPIOS and EPIOT definition 2/ Detailed neighbourhood search 1/ Envelopexchange x x x x x xx x x x x x x x x x x x o o o o o o o o o o o o EPIOS1 EPIOT1 x x x x x x x x x x x x o o o o o o o o o o o o EPIOS2 EPIOT2 x x x x x x x x x x Oasis4: communication (2/3) Parallel calculation of communication patterns in source PSMILe For each pair of source and target processes:

More Related