MARS Introduction and basic concepts

1. MARSIntroduction and basic concepts Computer User Training Course 2004 Carsten Maaß & Manuel Fuentes User Support, Data and Services Section

2. Contents • Introduction • Meteorological content • MARS language • MARS architecture • Retrieving data • Practicals

3. Introduction Meteorological Archival and RetrievalSystem • Meteorological data (GRIB: fields, BUFR: observations) • Large amount of data (size & number) • Operational & Research environment • Batch & interactive modes • Large number of users with different requirements: large datasets rarely ↔few fields very often • Heterogeneous environment

4. Introduction – MARS components • Client/Server architecture • Clients: workstations, supercomputers • Servers: supercomputers, dedicated servers • Several databases • Tape library

5. Introduction – Some figures • 760 TBytes of data in 6.7 • 109 fields • Growing daily by 750 GByte: • 250 GB Operational data • 500 GB Research • 100 000 requests (10 • 106 fields) per day • Analysis, forecast and observations since 1957 (ERA-40) • Operational forecast since 1985

6. Meteorological content – Atmospheric model Atmospheric model (T511 L60) • Analysis (synoptic hours: 00, 06, 12 and 18 UTC) • Surface • Model levels (60) • Pressure levels (21) • Isentropic levels • Forecast (10 day forecast based on 00/12 UTC Analysis) • 3 hourly steps from 3 to 72 hours • 6 hourly steps from 72 to 240 hours

7. Meteorological content – Wave model Wave model • Coupled with atmospheric model • Analysis (00, 06, 12 and 18 UTC) • Forecast (to 10 days) European Shelf Model (Mediterranean) • Analysis • Forecast to 5 days

8. Meteorological content – Boundary conditions Boundary Conditions Project (T511 L60) • Analysis (3dVar at 00, 06, 12 and 18 UTC) • Forecast (at 00/06/12/18 UTC to 96 hours)

9. Meteorological content – EPS Ensemble Prediction System (T255 L40 at 00/12 UTC) 50 different forecasts with perturbed initial conditions • Control forecast (to 21 days) • Perturbed forecasts (to 10 days) • Ensemble mean and standard deviation • Extreme forecast index • Forecast probabilities • Clusters • Tubes

10. Meteorological content – Other ensemble FC Seasonal Forecast • Atmospheric + Wave + Ocean models (00 UTC to 6 months) • Ensemble forecast techniques (40 members) • Different methods: with and without ocean data assimilation • Monthly means Monthly Forecast • Every two weeks (00 UTC to 32 days) • Atmospheric + Wave + Ocean models • Ensemble forecast techniques (50 members)

11. Meteorological content - Multi-Analysis Ensemble • Analyses from 4 Centres • NCEP • UK Met Office • Meteo-France • DWD • 5 Forecasts (to 10 days at 12 UTC): 4 centres + 1 consensus

12. Meteorological content – Monthly Means Averaged over each calendar month • Atmospheric • Analysis • Forecast • Wave • Analysis • Forecast

13. Meteorological content – Special datasets • Special Projects • ECMWF Re-Analyses (ERA-15, ERA-40) • DEMETER: Multimodel Ensemble for seasonal to Interannual prediction • PROVOST • ECSN-Hyretics • ECMWF Research experiments • Member State’s Research experiments • Member State’s own model data

14. Meteorological content – ERA-40 • ERA-40 period: Sep 1957 – Aug 2002 • Resolution: • Horizontal: T159L60, N80 (~1.125º) • Vertical: 60 ML, 23 PL, 15 PT, PV=±2 • Analyses (3dVar) at 00/06/12/18 • Forecasts at 06/18 to 6h and 00/12 to 36 h • Monthly means, vertical integrals, observations … • All Analysis data on-line • 1 file per 1 month of data

15. Meteorological content – Observations & Feedback • Observations • Surface data • Vertical soundings • Upper-air data • Satellite • Feedback • Analysis Input • Analysis Feedback

16. Meteorological content – Archive plans • Early delivery system • Multi-model Seasonal Forecast • Revised Monthly Forecast • Forecast satellite images

17. Meteorological content – Data formats WMO formats Fields in GRIB (GRid In Binary), ECMWF local extensions • Spherical Harmonics (upper-air fields, T511) • Gaussian Grid (surface fields, N256) • Latitude/Longitude (wave and ocean products) Observations in BUFR (Binary Universal Form Representation) • Instrument specific

18. MARS language Mechanism to name archived fields Request syntax: verb, parameter1 = value1, … = value2, parameterN = valueN • verb: action to be taken (e.g. retrieve, list, read) • parameter: predefined keyword (e.g. type, date, target) • value: value assigned to the parameter

19. MARS language • verb and parameter=value separated by commas, but last one • Spaces and tab characters are ignored • *, ! and # comment until end-of-line • Directives are not case sensitive • Values: predefined names, numeric values or strings (filenames) • Abbreviations: enough letters to uniquely identify keyword • Acronyms: usually initial letters of names • / is used as list separator → specify pathnames in quotes

20. MARS language – Retrieve request retrieve, action class = od, identification stream = oper, expver = 1, date = -3, date & time related time = 12, type = analysis, data related levtype = model levels, levelist = 1/to/60, param = temperature, grid = 2.5/2.5, post-processing target = “analysis”storage

21. MARS language – Identification of archive

22. MARS language - Date & time

23. MARS language – Fields

24. MARS language – Observations & images

25. MARS language – Storage Unix pathnames (using /) have to be enclosed in quotes, e.g. target = “/scratch/ms/gb/uid/analysis”

26. MARS language - Post-processing

27. MARS language – Execution control

28. MARS language – Values • Single value, predefined names, numbers, mnemonics param = temperature • List of values, separated by / step = 12/24/48 • Range of values, using keywords: to, / and by date = 20020101/to/20020131 step = 24/to/240/by/24

29. MARS language – Values • Expected number of fields is computed by multiplying number of values after expansion of ranges date = 20020101/to/20020131 31 fields • Certain parameters accept all as valid value levelist = all • Most parameters accept off as valid value levtype = surface, levelist = off • Not all possible combinations parameter = value name an archived field

30. Request examples - Re-Analysis Retrieval of snow depth from the ERA-40 archive for November 1993, for all analysis base times. It retrieves 120 fields. retrieve, class = e4, stream = oper, expver = 1, date = 19931101/to/19931130, time = 00/06/12/18, type = an, levtype = sfc, param = sd, target = “era40.199311.sd”

31. Request examples - Ensemble forecast Retrieval of surface temperature and 10-m wind components (U and V), 20 first members of the EPS for 2nd Jan 2001 for time steps 12, 36 and 60. It retrieves 180 fields. retrieve, class = od, stream = enfo, expver = 1, date = 20010102, time = 12, step = 12/36/60, type = pf, levtype = sfc, param = st/10u/10v, number = 1/to/20, target = “perturbed.sfc”

32. Request examples – Operational analysis Retrieval of sea surface temperature for first 10 days of May 2002, all synoptic times. It retrieves 40 fields. retrieve, class = od, stream = oper, expver = 1, date = 20020501/to/20020510, time = 00/06/12/18, type = an, levtype = sfc, param = sea surface temperature, target = “sst”

33. MARS Architecture • Client/Server • Protocol: MARS request • Clients, C program + libemos library (GRIBEX + Interpolation) • Supercomputers • Workstations and Servers • Applications like Metview (local / at ECMWF) • Remote client for Member States (security mechanism) • WebMARS • Data Server

34. MARS Architecture – Servers • Reports Database (RDB), on-line observations (for Operations only) • Fields Database (FDB) • Data produced by most recent cycles or experiments • Very fast access (on-line data) • Suitable for model input • Main Archives (4 servers) • Dedicated IBM servers, running AIX • 9 TB disk space • Tape management SW: HPSS • StorageTek silos

35. MARS Architecture - Request execution • Check syntax (MARS language and request syntax) • Print request to be processed • Query all Supercomputer’s FDB • Query main archives (if data not in FDB) • Transfer data • Post-processing while transferring (if needed) • Report on result

36. Retrieving data Request scheduling • Queue systemPriorities: user, request age, request cost (number of tapes and fields) Data collocation • MARS tree • Archive objects (for OD data) • 1 file per month of AN (1 level type, all times, levels, params) • 1 file per forecast (1 level type, all steps, levels, params) • 1 file per EPS (1 level type, all steps, members, levels, params)

37. Retrieving data - MARS tree

38. Retrieving data - Post-processing • Conversions • SH → SH (reduced truncation), GG, LL • GG (reduced) → GG (lower resolution or regular), LL • LL → LL (lower resolution) • Sub-area extractions (GG, LL, waves), reduces data volume • Derived fields (e.g. U and V from vorticity and divergence) • Rotation

39. Retrieving data - Post-processing Truncation before interpolation, reduces necessary resources

40. Retrieving data – Efficiency • Local Disk ($SCRATCH) • Estimate amount of data (list command) • Number of fields (up to tens of thousands / request) • Data size (up to several Gigabytes / request) • Check computer resources: quota, CPU time, … • Reduce number of tapes involved (better scheduling) • Retrieve as much data from the same tape as possible • Do not create unnecessary sub-archives

41. Retrieving data - Hints • Default values: minimize their use • No semantic check (only syntax is checked) • MARS messages • INFO request execution and report • WARNING unusual aspect of execution • ERROR system or data errors • FATAL terminates execution

42. Accessing MARS mars <<EOFretrieve, type = an, date = -1 target = “$SCRATCH/my_an”EOF • directives from input stream • directives from file cat > my_request <<EOFretrieve, type = an, date = -1 target = “$SCRATCH/my_an”EOFmars my_request

43. Additional resources • MARS documentationhttp://www.ecmwf.int/publications/manuals/mars/ • Web-MARSwww.ecmwf.int/services/archive/ • Data Services documentation www.ecmwf.int/products/data/archive/ • User Guide to ECMWF forecast products www.ecmwf.int/products/forecasts/guide/ • EMOSLIB (GRIBEX) Documentation www.ecmwf.int/publications/manuals/libraries/ • IFS Documentation www.ecmwf.int/research/ifsdocs/

44. Data Server – http://data.ecmwf.int/data/ • A standalone PC-Linux system outside the firewall • Public (non-commercial) distribution of data • Self-registration • Datasets • DEMETER, 24 GBytes • ERA-15, 1 GByte • ERA-40, 400 GBytes, 2.5º • Based on Web-MARS • Disk-only MARS server • MARS client, Metview, SMS • fastCGI, Perl, MySQL

45. Data Server • User interface • different for each dataset • built dynamically from MySQL contents • NetCDF is produced on-the-fly from GRIB (experimental) • Very popular since ERA-40 made available • About 1000 users • Download about 1TB/month • Useful tool for sharing data among project collaborators

46. Web-MARS – Archive catalogue • Content browsing of every field in the archive • Real-time (dynamic access to metadata) • Create MARS requests (without checking availability) • Check availability of data • Check for changes in the archive • “Cost” evaluation • Retrieval and plotting for few fields • URL based in MARS requests (can be edited) • More up to date than static content documentation

47. Web-MARS – Catalogue shortcuts Overview • Brings you directly into the catalogue • Static page to latest datasets • Suitable for new users Latest update • Page updated every night • Availability by source of data

48. Web-MARS – Data Finder • Allows to have different views of the archive • By period of time • By meteorological parameter • By data source (forecasting system) • Narrow the search for data • Availability • Brings you directly into the catalogue

49. Web-MARS – Changes in the archive • addition or discontinuation of fields Web-MARS – Parameter Database • GRIB table controlled view • Links to IFS documentation

50. Web-MARS – Server activity • Show archive activity • Monitor your requests • Learn how the queuing system works • Reason for queued requests