1 / 16

Changes at ECMWF

Changes at ECMWF. Walter Zwieflhofer Head of Computer Division. Operational system (October 2001). Data assimilation 4D-Var with a 12-hour data window Horizontal resolution: 120 km for inner and 40 km for outer loops Global atmospheric forecasts

ellema
Download Presentation

Changes at ECMWF

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Changes at ECMWF Walter Zwieflhofer Head of Computer Division

  2. Operational system (October 2001) • Data assimilation • 4D-Var with a 12-hour data window • Horizontal resolution: 120 km for inner and 40 km for outer loops • Global atmospheric forecasts • Forecasts to 10 days from 12 UTC at 40 km resolution • Ensemble forecasts to 10 days with 51 members at 80 km resolution • Four forecasts per day to 3 days to provide boundary conditions for some of the ECMWF Member States (40 km, 3D-Var) • Ocean wave forecasts • Global forecasts to 10 days at 40 km resolution • European waters forecasts to 5 days at 15 km resolution • Seasonal forecasts • Global forecasts to 6 months (ensemble with 30 members)

  3. N Hem Mean reduction in day-one error measured in pre-operational trials S Hem Forecasting-system changes since 1999 19r2 Increased resolution in stratosphere 21r1 Assimilation of raw TOVS/ATOVS radiances 21r4 New error statistics for 4D-Var Jb, SSMI wind data, corrected use of humidity observations, increased PBL resolution, improved cloud, convection and orography. 22r1 Improved SSM/I QC and second SSM/I satellite 22r3 New error variances (for Jb and Jo) additional ATOVS data, improved land-surface, sea-ice and LW radiation 23r1 12-hourly 4D-Var 23r3 T511/159 resolution, second ATOVS satellite, increased angular resolution in wave model

  4. Evolution of forecast skill for northern and southern hemispheres

  5. Error distribution for Day 5 (Europe)

  6. Annual-mean RMS errors against analyses from WMO scores

  7. Planned improvements • 2002 Increase of the inner loop resolution of 4D-Var from 120 km to 80 km • 2003-2004 Assimilation of cloud and rain in 4D-Var Higher vertical resolution in free troposphere and lower stratosphere Assimilation of AIRS data Operational running of a Severe Weather forecasting system • 2005-2006 Assimilation of IASI data

  8. Computer configuration – October 2001

  9. PrepIFS • ECMWF tool to prepare forecast and data assimilation experiments • Uses the operational configuration of the forecasting system and allows researcher to introduce modifications to: • input data (including the selection of observations) • the model code • the resolution to be used and many other parameters • Written Java, with a client part that can be installed on remote machines • Allows registered external users to configure and submit experiments via the Internet

  10. Major changes to thecomputing environment • Desktop systems (procurement completed in 2000) • Decision was taken to replace all proprietary UNIX workstations with PCs running LINUX over a period of three years • Data Handling System – DHS (procurement in progress) • A scalable system that will be able to support the expected growth over 6+ years is required • Invitation to Tender for a new system was issued earlier this year • Supercomputer – HPC (procurement in progress) • Fujitsu systems are rented and the contract ends on 31 Dec 2002 • Invitation to Tender for a replacement to cover the period 2003-2006 was issued in March this year

  11. DHS procurement • Archive holds currently some 400 TB of data (primary data) and has been growing by some 60% per annum • Two services need to be supported at ECMWF: • ECFS, a large-scale UNIX-like filesystem for general use • MARS, a customised meteorological archival and retrieval system • Vendors were asked to offer a scalable storage system and were given the option to offer a replacement for the ECFS and/or MARS software layer • DHS and HPC procurements were separated because: • Lifetime of the data handling systems tends to be considerably longer than that of the HPC systems • Allows a simpler HPC procurement not affected by the complexities associated with the replacement of the archive system

  12. HPC procurement • ECMWF asked for a phased introduction of resources covering the period 2003-2006, with a parallel run during the second half of 2002 • The specifications allowed for some flexibility, enabling vendors to adapt their response to their product development plans. This included the option to propose a 3-month shift of the service period • The funds available were published • Vendors were required to commit to performance increases, expressed as factors, over the performance of the Fujitsu systems installed at ECMWF • Requirements for disk space, I/O and network bandwidth were expressed as function of these factors

  13. The Benchmark Suite • IFS tests (for the performance commitments) • 4D-VAR (T511/255 L60; 2, 4 or 6 copies depending on PHASE) • DETERMINISTIC FORECAST (T799 L90; 2 or 4 copies) • EPS (T399 L60; 50 or 100 copies) • Burton Smith taxonomy: • Size * g >> 2 * O + L • Suitable for both T-type and C-type systems • IFS model runs well on all architectures • 4D-Var is very sensitive to communication bandwidth and has sections on which scalar machines do well • Resolutions representative for the workload planned for 2003-2006 were used

  14. Observations • Vendors were allowed to optimize the codes • However, changes that would affect the readability or portability were not permitted • Some vendors used a hybrid MPI / OpenMP approach • Some vendors found that such a hybrid approach did not improve the results obtained by using MPI only • Percentages of peak: • 4D-Var scaled not as well the deterministic forecast and the EPS • Scalar machines: 3% to 12%, depending on test/product • Vector machines: 14% to 38%, depending on test/product • Rule of thumb for ECMWF’s codes was confirmed: • ~10% for scalar architectures and ~30% for vector architectures

  15. Status of both procurements • The Tender Evaluation Boards have completed their work and have made recommendations to the ECMWF Director • The Director has awarded both contracts, subject to approval by ECMWF Council • Documents detailing the respective procurement process and the proposed solutions have been sent to the technical and financial committees of the ECMWF Council • These committees will meet in November to consider ECMWF’s proposals and to make recommendations to Council • Council will meet in December and (hopefully) authorize the Director to sign the appropriate contracts

  16. New HPC system New Network New Servers New DHS Changes over the next 18 months Linux PCs

More Related