Michael S. Fox-Rabinovitz , ESSIC (Earth System Science

1. 4th International Workshop on Next Generation Climate Models, for Advanced High Performance Computing Facilities 12-14 March, 2002, Boulder, CO Variable-Resolution SG-GCMs Michael S. Fox-Rabinovitz,ESSIC (Earth System Science Interdisciplinary Center, University of Maryland, and NASA/GSFC/DAO (Data Assimilation Office) foxrab@atmos.umd.edu or foxrab@dao.gsfc.nasa.gov Acknowledgments: L. L. Takacs, R. C. Govindaraju

2. Contents • INTRODUCTION:the variable-resolution stretched-grid (SG) approach to regional climate modeling • 2.SG-GCMANNUAL(Nov.-97 – Dec.98) simulation with the stretched grid withMULTIPLE (four) areas of interest • 3.A12-YEAR(Oct.-1986 – Dec.1998) AMIP-type SG-GCM simulation • LimitedENSEMBLESG-GCM integrations • Development of theNEW SG-fvGCMs with the NCAR CAM PHYSICS, andadvanced parallel computingtechniques • 6. Conclusions • 7. PROPOSALon the International Stretched-Grid Model Intercomparison Project(SGMIP)

3. Questions/hypotheses: • What are the advantages of the SG-approach vs. nested-grid • approach? • What are the computational problems for model dynamics to be • controlled when using the SG-approach? • What are the limitations/restrictions associated with the • SG-approach for regional climate modeling, in terms of necessity • to control the SG-parameters? • What is an optimal way for treating/calculating model physics and • stationary surface boundary forcing (orography and land-sea • differences): on a stretched or uniform intermediate resolution grid? • What is the potential of the SG-approach for obtaining: • * high quality/integrity of global circulation • * consistent interactions between global/large - and regional/meso- • scales • * efficient regional downscaling to mesoscales • * computational and parallel performance efficiency • Could variable resolution be used for future global climate models?

4. A STRETCHED GRID WITH ONE AREA OF INTEREST

5. An example of a stretched grid with one area of interest (AOI) with enhanced 60 km regional resolution over the U.S. A blow-up:

6. An example of a stretched grid with four areas of interest over: 1.the U.S./Western Atlantic ocean; 2. El Nino area/Brazil- Argentina; 3. India-China; nd 4. Eastern Indian Ocean/Australia

7. SG-design Q: What are optimal/necessary requirements/controls for a SG-design to be imposed for obtaining successful computationally efficientregional climate simulations that would provide: • An efficient downscaling over the area(s) of interest; • Consistent/realistic interactions between global/large- and regional/meso-scales; • A sufficient quality/skill of global circulation. • Fine uniform resolution over the area(s) of interest. • Uniform stretching with constant local stretching factors. • Moderate local stretching factors, within ~5% • Limited maximumgrid intervals, within 2-3 degrees, the resolution of typical GCMs(to control the accuracy glob.appr.)

8. Issues regarding two types of downscaling strategies Nested grid LAM driven by a GCM Variable Resolution GCM (SG-GCM) • Problems with one-way interactions • Problems with long-waves/global scales • Abrupt change of resolution generates noise and spurious phenomena • Need a buffer zone along the lateral boundaries to relax the driving and the nested variables • Non-autonomous. Depends on driving data for the LBCs • Spatial interpolations required • Vertical and temporal resolutions can be different • Climate solution depends on the domain size and resolution jump • Proved to be a workable solution • Consistent interactions between long-wave/global and regional/meso- scales • High qualityof global scales • No abrupt change of resolution, but stretching factor must be moderate • No buffer zone needed • Autonomous. No dependence on driving data for the LBCs • No spatial interpolation required • Vertical and temporal resolutions fixed for the whole globe • Climate solution depends on stretching • Proved to be a workable solution (e.g. Déqué 1995,1998; Fox-Rabinovitz et al. 1997,2000,2001; McGregor 1998) == ===

9. Variable-Resolution GCMs • U.S.: Finite-difference GEOS SG-GCM (Fox-Rabinovitz et al 1997, 2000, 2001, Fox-Rabinovitz 2000) • U.S.: Finite-volume SG-GCM (M. Fox-Rabinovitz and Kaosan Yeh, using the Lin-Rood GEOS fvGCM) • 3. Canadian RPN/CMC, GEM (Global Environmental Multiscale) model (Cote et al. 1998, 2000) • 4. Meteo-France, ARPEGE (Deque and Piedelievre 1995, Deque et al. 1998 Deque 2000) • 5. AustralianCSIRO Model (McGregor 1996, McGregor and Dix 1997, 1998); • The new SG-GCM (under development): • Spectral-elementSG-GCM (a companion study with F. Baer, J. Tribbia, A. Fournier) • Other variable-resolution models: • 7. University of Utah (UU) model (Paegle 1989, Paegle et al. 1996)) • 8. 2-D spectral model (Hardiker 1997, the FSU group) • 9. NCSU non-hydrostatic model (the Semazzi’s NCSU group)

10. GEOS (Goddard Earth Observing System) GCM • (the basic model) • Dynamics: finite-differences (Arakawa and Lamb 1981, Suarez and • Takacs 1995, Fox-Rabinovitz et al. 1991) • Filters: high latitude Fourier; Shapiro (1970); • Resolution: 2x2.5 or 1x1 degree, 70 or 48 layers; • Radiation: Chou and Suarez (1994); • Turbulence/PBL: Helfand and Labraga (1989); • Convection: RAS (Relaxed Arakawa-Schubert) Moorthi and • Suarez 1992; • Land-surface model: Koster and Suarez 1994; • Participation in AMIP

11. GEOS SG-GCM (Fox-Rabinovitz et al. 1997, 2000, 2001, 2002; Fox-Rabinovitz 2000) Same dynamics and physics as in GEOS GCM Model dynamics and orography calculated on the stretched grid Model physics calculated on an intermediate uniform resolution grid (Lander and Hoskins (1997)) The integration history resides effectively on the stretched grid Modified for variable-resolution polar (Takacs et al. 1999) and local filters Regional resolution for SGs (obtained by redistribution of global grid points of the 2x2.5 or 1x1 degree uniform grids): a) 60, 40, or 25 km with one large area of interest (like the U.S.) b) 50 or 40 km with four large areas of interest c) 10-15 km with one subregion of interest (like a river basin)

12. Development of the new SG-fvGCM (L-R dynamics, NCAR physics) (K. Yeh, M. Fox-Rabinovitz, S.J. Lin) * The SG-fv-dynamical core (dycore) has been developed and is being tested with the generalized Held-Suarez forcing * The generalized SG-fv-dycore is consistent with the original Lin-Rood (L-R) uniform-grid (UG) dycore of the NASA fvGCM * The dynamics and polar filter are modifies for variable resolution * The similarity of global fields is obtained for SG & UG dycores * Enhanced regional resolution provides efficient downscaling to mesoscales, i.e. it results in producing stronger gradients, stronger jets; and more pronounced mesoscale features over the region(s) of interest * The SG-dynamics (including orography) is combined with the intermediate resolution physics. Implementation of the SG-physics is being investigated. * Target resolution: for 4 or more areas of interest: 15-25 km; and for 1 to 4 areas of interest: 5-10 km In other words: at least doubling the resolution of any uniform grid

13. Parallelization techniques for the SG-GCMs 1. GEOS SG-GCM: - the MPI mode 2. SG- FV-GCM: - the MPI mode for distributed memory - the SMP mode for shared memory - the hybrid mode with multi-level parallelism, to take advantage of CPUs with shared memory Note: the overhead due to variable resolution is limited due to grid variability in the horizontal only

14. Variable-resolution grids used and planned to use:  • Used with finite-difference approximation:  • - Stretched grid with 1 area of interest • - Stretched grid with multiple (four) areas of interest • 2. Will be used with the new finite-volume SG-GCM • - Enhanced resolution over the model global orography • - Exploring new stretching strategies • Future tests: Using variable-resolution grids as an alternative to • uniform grids, for global simulation purposes: • - Stretched grids with multiple (6-16) areas of interest • - Global anisotropic monotonicgrids: • * flow-and-surface-boundary-forcing dependent • * population-and-pollution distribution dependent • * stationary adaptive grids, with gradually changing intervals • * dynamic adaptive grids slowly changing in time, i.e. same as • above but with added time adaptive capabilities

15. Simulation and data assimilation of the 1998 anomalous regional climate events with the GEOS SG-GCM and SG-DAS with multiple (four) areas of interest1. Quality/integrity of global fields2. Reproducing regional anomalies

16. 500 hPa KE spectra for the SG-GCM, SG-DAS, and ECMWF reanalyses for AMJ 1998

17. AMJ 1998 ZONAL U SG-GCM(50km) vs. SG-DAS(50km) analysis and ECMWF (1x1) reanalysis

18. GLOBAL PRECIP AMJ 1998 SG-GCM(50km) vs. GPCP(1x1) DATA

19. U.S. AMJ 1998 H-500 SG-GCM(50km) SG-DAS(50km) BIAS or (a) – (b)

20. U.S. AMJ 1998 H-500 ANOMALY SG-GCM(50KM) SG-DAS(50KM) ECMWF(1X1)

21. U.S. AMJ 1998 U-500 ANOMALY SG-GCM(50KM) SG-DAS(50KM) ECMWF(1x1)

22. U.S. AMJ 1998 PRECIP SG-GCM(50km) GPCP(1x1) NCEP(0.25x0.25) GAUGE DATA

23. Mexico MAM 1998 Precipitation SG-GCM(50km) SG-DAS(50km) Xie-Arkin(2x2) GAUGE and SAT. DATA

24. China July 1998 Precipitation SG-GCM(50km) SG-DAS(50km) Xie-Arkin(2.5x2.5) GAUGE DATA

25. China July 1998 Precip Anomaly SG-GCM(50km) vs. SG-DAS(50km) Xie-Arkin(2.5x2.5) GAUGE DATA

26. The12-year (1987-1998) SG-GCM simulation and limited ensemble calculations

27. 12-YEAR SIMULATION: 9-YEAR MEAN (1988-1996) JUNE PREC. SG-GCM(60KM) vs. U of DE GAUGE DATA and NCEP(Xie-Arkin) DATA

28. 12-YEAR SIMULATION 9-YEAR MEAN(1988-1996) JUNE PRECIP (for the SE U.S. subregion) SG-GCM(60KM) vs. U of DE GAUGE DATA and NCEP(Xie-Arkin) DATA

29. H-500 “Spaghetti” diagrams 12-year SG-GCM ensemble January 1987 January 1988 January 1989

30. H-500 Spaghetti diagrams 12-year SG-GCM ensemble July 1987 July 1989

31. Applications of SG-GCM and SG-DAS products: * Hurricanes, typhoons, and severe storms: - 25 km regional resolution - all the 1998 events, for all four global quadrants * Atmospheric chemistry transport experiments: - TOPSE - TRACE-P * Hydrologic model experiments for the major U.S. river basins (under development) * New satellite data impact studies (future plans)

32. Hurricane Floyd, 45-h forecast, 9/13/1999, SG-GCM(0.25X0.25) (low-level clouds; SLP) (wind-speed; RH)

33. Hurricane Floyd, 84-h forecast, 9/15/1999, SG-GCM(0.5X0.5) (SLP(lat 30N); wind-speed; omega) (U-wind; V-wind; SLP)

34. TOPSE Mission (FM-2000) CURTAIN PLOTS (FLIGHT TRACK: DASHED LINE) TRACE GAS: 7Be UNIFORM GRID (2.5x2.5) STRETCHED GRID (60KM) (the courtesy of D. Allen and K. Pickering)

35. Recent Journal Publications on the GEOS SG-GCM and SG-DAS • Fox-Rabinovitz, M. S., G.L. Stenchikov, M.J. Suarez, and L.L. Takacs, 1997: A Finite-Difference GCM Dynamical Core with a Variable Resolution Stretched Grid. Mon. Wea. Rev.,vol. 125, No. 11, 2943-2968. • Qian, J.-H., F. Giorgi, and M. S. Fox-Rabinovitz, 1999: Regional Stretched Grid Generation and its Application to the NCAR RegCM. JGR, v. 104, NO. D6, 6501-6513. •  Fox-Rabinovitz, M. S., G.L. Stenchikov, M.J. Suarez, L.L. Takacs, and R.C. Govindaraju, 2000: A Variable Resolution GCM Dynamical Core with Real Orography. Mon. Wea. Rev., Vol. 128, No. 6, 1883-1898. •  Fox-Rabinovitz, M. S., 2000: Regional climate simulation of the anomalous U.S. summer events using a variable-resolution stretched-grid GCM. J.Geophys Res.(the special issue on Advances in Regional Climate Modeling), Vol. 105 , No. D24 , pp. 29,635-29,646. • Fox-Rabinovitz, M. S., L.L. Takacs, and M.J. Suarez, 2001: A Variable Resolution Stretched Grid GCM: Regional Climate Simulation. Mon. Wea. Rev., Vol. 129, No. 3, pp. 453-469.

36. Recent Journal Publications on the GEOS SG-GCM and SG-DAS (cont.) • 6. Takacs, L. L., W. Sawyer, M. J. Suarez, and M. S. Fox-Rabinovitz, 1999: Filtering Techniques on a Stretched Grid GCM, NASA/TM-1999-104606, Vol. 16, 50 pp. • 7. Berbery E. H., and M.S. Fox-Rabinovitz, 2002: Simulation of the North American Monsoon System and its Link to Midwest Precipitation using a variable resolution stretched grid GCM, submitted. • 8. Fox-Rabinovitz, M.S., L.L. Takacs, and R.C. Govindaraju, 2002: A variable-resolution SG-GCM and SG-DAS with multiple areas of interest: Studying the anomalous regional climate events of 1998, submitted. • Fox-Rabinovitz, M.S., L.L. Takacs, and R.C. Govindaraju, 2002: A 12-year • AMIP-type SG-GCM simulation of the U.S. climate, to be submitted. • Yeh, K.-S., M.S. Fox-Rabinovitz, and S.J. Lin, 2002: A variable-resolution • finite-volume dynamical core, to be submitted. • 11. R.J. Park, K.E. Pickering, D.J. Allen, G.L. Stenchikov, and Fox-Rabinovitz, M.S.,: 2002: Simulation of tropospheric ozone using the University of Maryland chemical transport model. Part II: Regional simulation using a stretched-grid formulation, to be submitted.

37. Conclusions • The variable-resolution stretched-grid GCMs with enhanced resolution over the area(s) of interest are viable tools for regional climate simulation, with an advantage of providing efficient regional downscaling to mesoscales while preserving the high quality of global circulation. • 2. The successful annual SG-GCM simulation of the anomalous • regional climate events of 1998 with 50 km regional resolution are • performed for FOUR areas of interest (one at each global • quadrant): 1. the U.S. and Mexico; 2. India- China; 3. Brazil- • Argentina; and 4. Eastern Indian Ocean and Australia. • 3. The successful12-yearSG-GCM AMIP-type simulation is • performed for the U.S. The applicability of the SG-approach to • long-term multiyear regional climate modeling is confirmed. • The limited SG-GCMensembleintegrations are found useful forreducing the uncertaintyof regional climate simulation.

38. Conclusions (cont.) • 5. Development of the new GEOS SG-fvGCM employing finite-volume (Lin-Rood) dynamics and NCAR CCM3/4 physics. • - The new stretching strategies are being investigated. • - Using anisotropic monotonic (gradually and slowly) adaptive (stationary and dynamic) grids as an alternative to uniform grids • will be addressed in the future. • Continuationof the collaborative studies on application of the • SG-GCM simulated products for driving tropospheric chemistry • transport experiments • 7.The SG-modeling has matured enough for initiating the internationalSGMIP with participation of major centers and groups from the U.S., Canada, France, and Australia. The first stage is planned for 2002-03. The scope of the project (including ensembles) depends on the computer time available. SGMIP will be connected to AMIP-II at the next stage.

39. SGMIP (Stretched-Grid Model Intercomparison Project) with • participation of the major centers/groups from the U.S. (NASA/GSFC/DAO and UMD), Canada(RPN/CMS), France (Meteo-France), and Australia (SCIRO) (started). • Performing ensemble integrations (started). • Developing extensive applications: providing forcing products for high resolution atmospheric chemistry, hydrologic, and other models (started or planned). • Developing more flexible/sophisticated SG-generators (planned). • Developing model physics for mesoscale resolution (a long-term effort). • Developing new GCMs with flow-dependent anisotropic grids for both regional and global and climate simulations (a long-term effort). Future Research Areas

40. Proposal on the Stretched-Grid Model Intercomparison Project (SGMIP) The variable-resolution SG-GCMshavematured enoughto justify theintercomparison effort • There are several SG-GCMs with different dynamics and physics used for regional climate simulation and applications. The obtained results have shown the feasibility of the SG-modeling approach.  • There are several computational topics that will be addressed by the intercomparison:  • Different stretchingstrategies • Different approximations of model dynamics, and filters • Different model physics including treatment of physics on stretched grids • Ensemble calculations • Optimalperformance on parallel supercomputers

41. Proposal on SGMIP (cont.) • The 12-year simulation period (1987-1999) including the ENSOcycles is proposed for:  • Studying the impact of resolution on efficient/realistic downscaling to mesoscales • Studying the U.S. anomalous regional climate events (floods, droughts, etc.) and major monsoonal circulations in the context of ENSO cycles at mesoscale resolution • Studying the impact of surface boundary condition forcing • Studying the U.S. water and energy cycles • Studying the impactof a multi-model ensemble

42. Proposal on SGMIP (cont.) • The representatives of major centers/groups have expressed their commitment in participating in the project. • The participating centers/groups are: NASA/GSFC (Fox-Rabinovitz); • Meteo-France (Deque); RPN/CMS, Canada (Cote and Dugas); CSIRO Australia (McGregor); UMD (Baer); NCAR (Tribbia and Fournier). • Five SG-GCMs will be included into the first stage of the project • (2002-2003). Two-four other SG-GCMs will join at the next stage. • The AMIP representatives (L. Gates and P. Gleckler) have expressed their interest in this effort to become in the future a special regional project within AMIP. • The proposal was discussed at the meeting held during the Workshop on Regional Climate Research, April 2001, Boulder, CO with all the participants. The consensus was that we have to go ahead with the project. • The details and logistics have been discussed and finalized. The requirements and support for the intercomparison project have been specified in June, 2001. The computing resources for the U.S. participants are being identified.