Outline

2. Outline • OASIS historic • OASIS community today • PRISM first coupler: Oasis3 • model adaptation • coupled model configuration • communication • interpolations/transformations • PRISM final coupler: Oasis4 • model adaptation • the Driver • communication • interpolations/transformations • in the prototype • still to be done • Conclusions

3. OASIS historic OASIS:OceanAtmosphereSeaIceSoil • Developed since 1991 in CERFACS to couple existing GCMs. At the time: • Models at relatively low resolution (~10000-20000 pts) • Small number of 2D coupling fields (~10) • Low coupling frequency (~once/day) • flexibility was very important, efficiency not so much!

4. OASIS historic • 1993: OASIS 1.0 (CRAY pipes & files) • 1995: OASIS 2.0 (PVM3): Heterogeneous coupling (Cathode) • 1996: OASIS 2.1: Global warning scenario (SIDDACLICH) • 1998: OASIS 2.2 (SIPC): Seasonal forecasting (Demeter) • 1999: OASIS 2.3 (GMEM - NEC): Intercomparison (SINTEX) • 2000: OASIS 2.4 (MPI2): Climate Natural variability (Predicate) • 2003: OASIS 3 (MPI2-MPI1 + PSMILe + I/O): PRISM • => new Oasis3 (oasis3_prism_2-1) now available !

5. OASIS community today • CERFACS (France) ARPEGE3 - ORCA2LIM ARPEGE3 - OPA 8.1 ARPEGE3 - OPAICE • METEO-FRANCE (France)ARPEGE4 - ORCA2 ARPEGE medias -OPAmed ARPEGE3 - OPA8.1 ARPEGE2 - OPA TDH • IPSL- LODYC, LMD, LSCE (France) LMDz - ORCA2LIM LMDz - ORCA4 LMDz - OPA ATL3/ATL1 IFS - OPA 8.1 ECHAM4 - ORCA2 • MERCATOR (France) PAM(OPA) • MPI - M&D (Germany) ECHAM5 - MPI-OM ECHAM5 - C-HOPE PUMA - C-HOPE EMAD - E-HOPE ECHAM5 - E-HOPE ECHAM4 - E-HOPE • ECMWF(UK)IFS Cy23r4 - E-HOPE IFS Cy15r8 - E-HOPE

6. OASIS community today • CGAM-Reading (UK) HadAM3 - ORCA2 • Southampton University (UK) Inter. Atm - OCCAM lite • UCL (Belgium) LMDz - CLIO • SMHI (Sweden) ECHAM - RCA RCA(region.) – RCO(region.) • U. of Bergen (Norway) MM5 - ROMS • KNMI (Netherlands) ECHAM5 - MPI-OM • DMI (Danemark) ECHAM - HIRLAM • INGV (Italy) ECHAM5 – MPI-OM • IRI (USA) ECHAM4 - MOM3 • JAMSTEC (Japan) ECHAM4 - OPA 8.2 • BMRC (Australia) BAM - MOM4 BAM3 - ACOM2 • U. of Tasmania (Australia) Data Atm. - MOM4 • CAS,IIT Delhi (India) MM5 - POM

7. Static coupling: all coupling parameters defined initially. PRISM first coupler: Oasis3 • Mono-process coupler + parallel coupling library (PSMILe) • synchronisation of the component models • coupling fields exchange • I/O actions • mono-process interpolation • Flexibility, modularity: • Arbitrary number of models and coupling fields • Different coupling frequencies for each field • Different transformations for each field

8. Oasis3: model adaptation (1/3) PRISM System Model Interface Library (PSMILe) API : • Initialization: • call prism_init_comp(comp_id, comp_name, ierr) • Retrieval of component local communicator: • call prism_get_localcomm(local_comm, ierr) • Global grid definition (master process only) • call prism_write_grid(grd_name, nx, ny, lon, lat) • call prism_write_corner(grd_name, nx, ny, nc, clon, clat) • call prism_write_mask(grd_name, nx, ny, mask) • call prism_write_areas(grd_name, nx, ny, area)

9. Oasis3: model adaptation (2/3) PRISM System Model Interface Library (PSMILe) API : • Local partition definition (by each process in the global index space) • call prism_def_partition(part_id, ig_paral, ierr) • Coupling or I/O field declaration(name, partition id, status, shape, type) • call prism_def_var • (var_id, var_name, part_id, var_nbrdims, var_inout, • var_shape, var_type, ierr) • End of definition • call prism_enddef(ierr)

10. Oasis3: model adaptation (3/3) PRISM System Model Interface Library (PSMILe) API: : • Coupling or I/O field sending and receiving: • in model time stepping loop • depending on user’s specifications in namcouple: • user’s defined source or target (end-point communication) • coupling or I/O sending or receiving at appropriate times • automatic averaging/accumulation • automatic writing of coupling restart file at end of run • call prism_put (var_id, time, var_array, ierr) • call prism_get (var_id, time, var_array, ierr) • Termination: • call prism_terminate (ierr) • Auxiliary routines: • call prism_put_inquire(var_id, time, info) • call prism_put_restart(var_id, time, info)

11. Oasis3: coupled model configuration • In text file namcouple, read by Oasis3 main process, and distributed to component model PSMILes at beginning of run: • total run time • component models • number of coupling fields • for each coupling field: • source and target names (end-point communication) (var_name) • grid acronym (grid_name) • coupling and/or I/O status • coupling or I/O period • transformations/interpolations

12. O A O A Oasis3 O • Parallel communication between parallel models and Oasis3 interpolation process A O B A B A • Direct communication between models with same grid and partitioning B A A file • I/O functionality (automatic switch between coupled and forced mode): GFDL mpp_io library A A Oasis3: communication • PSMILe based on MPI1 or MPI2 message passing • End-point communication; one-to-one

13. Oasis3 • separate sequential process • neighbourhood search • weight calculation • interpolation per se during the run O A O A Oasis3 O A O Oasis3: interpolations/transformations • on 2D scalar fields only • Interfacing with RPN Fast Scalar INTerpolator package • nearest-neighbour, bilinear, bicubic for regular Lat-Lon grids • Interfacing with SCRIP1.4 library • nearest-neighbour, 1st and 2nd order conservative remapping for all grids • bilinear and bicubic interpolation for «logically-rectangular» grids • Bilinear and bicubic interpolation for reduced atmospheric grids • Other spatial transformations: flux correction, merging, etc. • General algebraic operations

14. Need to optimise and parallelise the coupler PRISM final coupler: Oasis4 • Situation today: • Higher resolution, parallel and scalable models • Higher coupling frequencies desirable • Higher number of models and (3D) coupling fields • Massively parallel platforms • Oasis4 is composed of: • a Driver • a Transformer • a new PRISM System Model Interface Library

15. PRISM final coupler: Oasis4 • Some keynotes: • Developers: • Sophie Valcke (CERFACS) - Design, XML input • Damien Declat (CERFACS) - Driver, transformer, XML input • René Redler (NEC CCRLE) - PSMIle, portability to Linux PC cluster • Hubert Ritzdorf (NEC CCRLE) - Parallel search, transformer interaction • Reiner Vogelsang (SGI) - I/O, portability to SGI • Th. Schoenemeyer (NEC HPCE) - Parallelisation, portability to NEC SX • Jean Latour (Fujitsu ) - Portability to Fujitsu VPP and PC cluster • Public domain; open source license (LGPL) • All external libraries and modules used exist in public domain versions; vendor optimized versions may also exist: • MPI1 and/or MPI2 • mpp_io; NetCDF/parallel NetCDF • libXML • SCRIP • Programming language: Fortran 90 and C

16. Oasis4: model adaptation (1/5) New PSMILe API: • Application initialization (1 call/application) : • call prism_init • (appl_name, ierr) • Component application (1 call/component) : • call prism_init_comp • (comp_id, comp_name, ierr) • Retrieval of component local communicator (1 call/component): • call prism_get_localcomm • (comp_id, local_comm, ierr) • Inquire SCC or SMIOCs persistent values: • call prism_get_persist (…)

17. Oasis4: model adaptation (2/5) • Definition of grid (3D) • call prism_def_grid • (grd_id, grd_name, comp_id, grd_shape, type, ierr) • call prism_set_corners • (grd_id, nbr_crnr, crnr_shape, crnr_array, ierr) • Placement of scalar points, vector points, subgrids on the grid: • call prism_set_points • (pt_id, pt_name, grd_id, pt_shape, pt_lon, pt_lat, pt_vert ,ierr) • call prism_set_vector • (vct_id, vct_name, array_of_pt_ids,ierr) • call prism_set_subgrid • (sgrd_id, sgrd_name, grd_id, n_sgrd, sgrd_shape, sgrd_array, ierr) • Function overloading to keep the interface concise and flexible

18. Oasis4: model adaptation (3/5) • Definition of grid cell dimension • call prism_set_scalefactors • (grd_id, scale_shape, scale_array, ierr) • Definition of grid cell mask • call prism_set_mask • (msk_id, grd_id, msk_shape, msk_array, ierr) • Definition of point local coordinate system angle • call prism_set_angle • (pt_id, angle_shape, angle_array, ierr)

19. prism_set_vector Oasis4: model adaptation (4/5) prism_def_grid Example prism-set-corners prism_set_points prism_set_points prism_set_points prism_set_mask

20. Oasis4: model adaptation (5/5) • Coupling or I/O field declaration • call prism_def_var • (var_id, var_name, grd_id, pt_id, msk_id, var_nbrdims, • var_shape, var_type, ierr) • End of definition • call prism_enddef(ierr) • Coupling or I/O field sending and receiving: • in model time stepping loop • depending on user’s specifications in SMIOC: • user’s defined source or target, component or file (end-point communication) • coupling or I/O sending or receiving at appropriate times • automatic averaging/accumulation • automatic writing of coupling restart file at end of run • call prism_put (var_id, date, date_bounds, var_array, info, ierr) • call prism_get (var_id, date, date_bounds, var_array, info, ierr)

21. Oasis4: coupled model configuration (1/2) • Application and component description (XML files): • For each application (code): one Application Description (AD): • possible number of processes • components included • For each component in the application: one Potential Model Input and Output Description (PMIOD) • component general characteristics: name, component simulated, … • code general characteristics: lab, documentation, contact, … • grid information: domain, resolution(s), grid type, … • potential I/O or coupling variables: • local name, standard name • units, valid min and max • numerical type • associated grid and points • intent –input and/or output • persistent variables: name, units, numerical type, possible values, …

22. Oasis4: coupled model configuration (2/2) • Coupled model configuration (XML files): The user produces • a Specific Coupling Configuration (SCC): • experiment and run start date and end date • start mode (MPI1, MPI2) • applications, components for each application • host(s), number of processes per host, ranks for each component • For each component, • a Specific Model Input and Output Configuration (SMIOC) • PMIOD component and code general characteristics • grid information: chosen resolution, … • I/O or coupling variables: • local and standard name, units, valid min max, numerical type, grid • activated intent –input and/or output • source and/or target (component and/or file) • coupling or I/O dates • transformations/interpolations/combinations • persistent variables: name, units, numerical type, chosen value, …

23. ATM-LAND AD Driver ATM PMIOD V1: in, metadata V1 V2: out, metadata V2 V3: out, metadata V3 user user OCE AD user OCE PMIOD V1: out, metadata V1 V2: in, metadata V2 user • SCC • ATM:... • OCE:... • LAND:... OCE OCE SMIOC V1 : to ATM, T1 to file V1 V2 : from ATM, T2 ATM ATM SMIOC V1 : from OCE, T1 V2: to OCE, T2 V3 : to LAND T LAND V2 V2 V3 V1 V1 LAND PMIOD V3: in, metadata V3 V4: in, metadata V4 user V4 fileV1 fileV4 Definition Phase LAND SMIOC V3 : from ATM V4 : from fileV4 Composition Phase Deployment Phase

24. Oasis4: the Driver Driver • Manages a static coupling: • all applications are started directly in the run script (MPI1 mode) or launched by the Driver at the beginning of the run (MPI2 mode) for the whole run. • Participates to communicator set-up • Reads the SCC and SMIOcs information and distributes it to the component PSMILes

25. C C C C C C OB OB O2 T O1 OB OB C C O2 O1 OB OB C C • Different grid and decomposition • via Transformer: • for each target point, parallel calculation of source interpolation neighbours in source PSMILe • Same grid, different decomposition • direct repartitioning: • for each target point, parallel calculation of source matching point in source PSMILe Oasis4: communication (1/3) • More elaborate PSMILe based on MPI1 or MPI2 • Parallel communication including repartitioning: • based on geographical description of the partitions • parallel calculation of communication patterns in source PSMILe

26. x x x x x x x x x x x x x x x x x x x x o o o o o o o o o 3/ EPIOS and EPIOT definition 2/ Detailed neighbourhood search 1/ Envelopexchange x x x x x xx x x x x x x x x x x x o o o o o o o o o o o o EPIOS1 EPIOT1 x x x x x x x x x x x x o o o o o o o o o o o o EPIOS2 EPIOT2 x x x x x x x x x x Oasis4: communication (2/3) Parallel calculation of communication patterns in source PSMILe For each pair of source and target processes:

27. file A file1 A A file A A A file2 A A A file3 single distributed parallel Oasis4: communication (3/3) • parallel 3D neighbourhood search in each source process PSMILe • new search algorithm: parallel multigrid technique, higher efficiency • extraction of useful part of source field only • end-point communication • one-to-one or one-to-many; combination • source data can be consumed by more than one target model or file and at a different frequencies • parallel I/O (vector, bundles, vector bundles): GFDL mpp_io, parNetCDF

28. OB C C C O2 OB T C O2 OB C Oasis4: interpolations/transformations • PSMILe: • time transformations, local transformations • global conservation, combinations • Transformer: • loops over PSMILe requests • EPIOS and EPIOT definition(source PSMILe) • For each EPIOT point, neighbours in EPIOS (source PSMILe) • =>Transformer calculates interpolation weights • Prism_put of EPIOS (source PSMILe) • =>Transformer performs interpolation per se • Prism_get of EPIOT (target PSMILe) • well suited for MPI parallelisation

29. Oasis4: interpolations/transformations • Different interpolation schemes: • nearest-neighbour, bilinear,bicubic, conservative remapping, … On different types of fields • 1D-2D-3D scalar, vector, bundles, bundles of vectors On different types of grids • blockstructured (regular and irregular) • unstructured • static or dynamic

30. Oasis4: in the prototype (03/2004) • Access of XML information • Coupling and I/O of n parallel applications with m components. • Coupling exchange with repartitionning, direct or through the (mono-process) Transformer • Interpolations: • PSMILe (non-exact) parallel neighborhood search • 3D nearest-neighbour • I/O: • single and parallel mode • Coupling and I/O exchange from one source to many targets • Local transformations (scatter, gather, add or mult scalar, statistics) • Basic time transformation (average, accumul, min, max)

31. Oasis4: still to be done • Completion of PSMILe calendar routines (internal and external) • PSMILe API for model access to SCC and SMIOC information • Interpolation: • More schemes (conservative, 3D, etc.) • Exact parallel neighbour search • Transformer parallelisation • Field reduction, combination • Full support of vector and bundle fields (I/O OK) • I/O: distributed mode (parNetCDF) • Adaptative grids • Unstructured grids

32. Conclusions • Calendar: • Oasis3_prism_2-1 available • Oasis4 prototype available within days • Oasis4 final PRISM version due 12/2004 • Follow-on: • Oasis4 developments will go on after 12/2004 with PRISM sustained support • FP6 funding (march 2005)? • Static vs dynamic couplers: • In the near and mid future, Oasis4 and Palm (Cerfacs dynamical coupler) will remain separate software • Fusion may be considered when climate modelling and data assimilation communities will join

33. The end