1 / 45

An Introduction to GRIB2

An Introduction to GRIB2. Simon Elliott EUMETSAT simon.elliott@eumetsat.int. Saturday Afternoon Agenda. Background (i.e. my background!) Overview (codes, GRID, GRIB, GRIB2) Code Structure (Sections like others, iterations of sections). GRIB2 Code Templates (what kinds are there … examples)

denisemartin
Download Presentation

An Introduction to GRIB2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Introduction to GRIB2 Simon Elliott EUMETSAT simon.elliott@eumetsat.int RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  2. Saturday Afternoon Agenda • Background (i.e. my background!) • Overview (codes, GRID, GRIB, GRIB2) • Code Structure (Sections like others, iterations of sections) • GRIB2 Code Templates (what kinds are there … examples) • Tables (Template tables, code and flag tables, examples) • Identification of parameters (?) • GRIB2 compression (methods, explanations of some) RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  3. Sunday Morning Agenda • Procedure for code form update (addns to tables, addition of tables/templates, new editions) • Changes not requiring code change (adding table/template entries for decoders … down stream code change) • Changes requiring code change (always for data provider, plus new editions) • Case studies of Cloud Mask and Precipitation Data (bespoke Encoder and Decoder, example product) • GRIB2 Interface for NMC (example of NCEP code) • Review/Discussion RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  4. Overview • Intention is to share information efficiently and unambiguously • Data should be accessible by anyone, not encrypted • Format should allow all information to be included (not rounded or skipped) • Format should allow addition of new data types • WMO “codes” developed per data type, TEMP, SYNOP, SATOB, et c. FM 35 TEMP FM 88 SATOB FM 87 SARAD FM 86 SATEM FM 85 SAREP FM 42 AMDAR FM 41 CODAR FM 13 SHIP RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  5. GRID, GRIB, GRIB2 • GRID was developed for gridded data (forecast, analysis) • ASCII -> human readable? • ASCII -> big files, slow transfer • Symbolic letters and code tables: F1F2NNN in Section 0 • F1F2 is originating centre as per C-1, • NNN is catalogue number of grid used by centre • GRIB introduced for binary exchange (storage and transmission efficiency). • Edition 0, 1985 • Edition 1, 1990 • GRIB Edition 2 (i.e. GRIB2) because Code Table 2 was full, EPS data, ... RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  6. GRIB1 structure Section 0 Indicator section: “GRIB” Section 1 Product definition section: unique standard template Section 2 Grid description section: one of the standard templates describing a type of grid Section 3 Bit map section Section 4 Binary data section Section 5 End section: “7777” GRIB uses the concept of Template:« Description of the standardized layout of a set of data items » RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  7. On the GTS By the WAFS Centres (« ICAO » products) For archiving of fields on MDD GRIB usage Main limitation is: • One parameter on one level for one grid per GRIB field RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  8. Transmission and archiving of: Spectral data Multi-dimension data Long-range and climate products Ensemble products (EPS) Also: No convention for missing data IEEE not used for floating point data No support for small time steps No cross-sections, no time-sections No Hovmöller Diagrams (ex. Time-longitude) Limited support for images GRIB1 weaknesses RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  9. Modularity: code and parameter tables referred to through templates Flexibility: new tables and templates can be added ALSO: All GRIB1 fields can be described in GRIB2 More compression schemes (e.g. introduction of JPEG 2000 and PNG) Need for “object oriented” GRIB2 (more modules = modularity) RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  10. Experimental GRIB2 presented at CBS 98 Finalized in Spring 2000 Approved by CBS in Autumn 2000 Operational in November 2001 First products in 2003 GRIB2 calendar RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  11. Generic structure • Identification: GRIB/BUFR/CREX • Header: Date, time, originator, table versions ... • Optional section: Metadata (potentially XML), private data … • Data description: What sort of data follows • Actual data: here • Closure: “7777” Table driven codes generally have this structure  RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  12. GRIB2 structure • Section 0 Indicator section: “GRIB” • Section 1 Identification section: data characteristics • Section 2 Optional (local) section: anything or nothing • Section 3 Grid definition section: geometry of grid used, potential reference to pre-defined grid • Section 4 Product definition section: description of data type, e.g. satellite data from spectral bands xx, yy, zz • Section 5 Data representation section: packing method used for data, reference value, scale factor et c. • Section 6 Bit map section: data present indicators (if used) • Section 7 Data section: data values themselves • Section 8 End section: “7777” Note: Sections 1 to 7 start with section length and number RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  13. Section 0 Section 0: Indicator section • Octets 1 - 4: “GRIB” in ASCII (i.e. 71 82 73 66) • Octets 5 - 6: Reserved (normally set to 0) • Octet 7: Discipline (CT 0.0, e.g. 3 for space) • Octet 8: GRIB edition number (2) • Octets 9 - 16: Total length in octets Messages of length up to ~18x1018 (or 264 - 1) bytes, i.e. ~18,000,000 Tb RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  14. Section 1 Section 1 Identification section • Octets 1 - 4: Section length • Octet 5: Section number (1) • Octets 6 - 9: Originating centre and sub-centre • Octets 10 - 11: Master and local table version numbers • Octet 12: Time significance (CT 1.2, e.g. 3 for observation time) • Octets 13 - 19: Date / time • Octet 20: Production status of data (CT 1.3, e.g. 0 for operational data) • Octet 21: Type of data (CT 1.4, e.g. 7 for processed radar observations) RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  15. Section repetition GRIB2 allows some groups of sections to be repeated for efficiency (not in GRIB1, one field per message). Non-repeated sections stay in effect • Section 0 (Indicator) • Section 1 (Identifier) • Section 2 (Local use) • Section 3 (Grid) • Section 4 (Product) • Section 5 (Data Repn) • Section 6 (Bit map) • Section 7 (Data) • Section 8 (Ending) Repeatable Repeatable Repeatable RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  16. Section repetition (example 1) • Section 0 (Indicator) • Section 1 (Identifier) • Section 3 (Grid) • Section 4 (Product) • Section 5 (Data Repn) • Section 6 (Bit map) • Section 7 (Data) • Section 4 (Product) • Section 5 (Data Repn) • Section 6 (Bit map) • Section 7 (Data) • Section 4 (Product) • Section 5 (Data Repn) • Section 6 (Bit map) • Section 7 (Data) • Section 8 (Ending) Forecast from same centre on same grid for same parameter at same level but from but for different validity times Validity time is in Section 4 (Product Definition) Section 4 to 7 repeated in message 0600Z 1200Z 1800Z RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  17. Section repetition (example 2) • Section 0 (Indicator) • Section 1 (Identifier) • Section 3 (Grid) • Section 4 (Product) • Section 5 (Data Repn) • Section 6 (Bit map) • Section 7 (Data) • Section 3 (Grid) • Section 4 (Product) • Section 5 (Data Repn) • Section 6 (Bit map) • Section 7 (Data) • Section 3 (Grid) • Section 4 (Product) • Section 5 (Data Repn) • Section 6 (Bit map) • Section 7 (Data) • Section 8 (Ending) Forecast from same centre for same validity time for same parameter at same level but from but on different grids Grid is defined in Section 3 (Grid Definition Section) Section 3 to 7 repeated in message GRID A GRID B GRID C RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  18. Grid definition templates • Grid definition is in Section 3 (GRIB1, Section 2) • Can refer to centre’s own grid or use Grid Definition Template (GDT) to specify details • Many GDTs have been developed, e.g. • 3.0: Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) • 3.1: Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) • 3.2: Stretched Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) • 3.3: Stretched and Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) • 3.20: Polar Stereographic • 3.90: Space view perspective, or orthographic RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  19. A grid definition template example GDT 3.90 - Space view perspective or orthographic Floating point numbers stored in 5 bytes as scale factor (one byte) and scaled value (4 byte IEEE floating point) RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  20. GDT example continued Repeating sections 4 to 7 means this information need not be repeated for multiple fields in one GRIB2 message RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  21. Product definition templates • Product definition is in Section 4 (GRIB1, Section 1) • Many PDTs have been developed, e.g. • 4.0: Analysis or forecast on horizontal level or layer at a point in time • 4.1 to 4.4: Various information pertaining to ensemble forecast systems • 4.5: Probability forecast on horizontal level or layer at a point in time • 4.7: Analysis or forecast error on horizontal level or layer at a point in time • 4.20: Radar products • 4.30: Satellite products • Hybrid sigma levels can be specified in Section 4 RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  22. A product definition template example PDT 4.30 - Satellite products RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  23. PDT example continued Note: Octets 25 to 24 are given for band 1, but are repeated for each contributing band up to NB, as specified in octet 14 RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  24. Code table 4.3: Type of generating process Code figure 0 1 2 3 4 5 6 7 8 192 - 254 255 Meaning Analysis Initialisation Forecast Bias corrected forecast Ensemble forecast Probability forecast Forecast error Analysis error Observation Reserved for local use Missing RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  25. Data representation templates • Data representation is in Section 5 (GRIB1, within Section 4) • Some DRTs have been developed, e.g. • 5.0: Grid point data - simple packing • 5.1: Matrix values at grid point - simple packing • 5.2: Grid point data - complex packing • 5.3: Grid point data - complex packing and spatial differencing • 5.50: Spectral data - simple packing • 5.51: Spherical harmonics data - complex packing • Section 5 also gives total number of data values to be found in Section 7 RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  26. A data representation template example DRT 5.0 - Grid point data - simple packing IEEE floating point numbers in 32 bits: seeeeeee emmmmmmm mmmmmmmm mmmmmmmm RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  27. Simple and complex packing • Simple packing • Y x 10D = R + (X1 + X2) x 2E • Y is original value, D is decimal scale, R is reference, E is binary scale, X1 is 0, and X2 is encoded value • In template R, E, D and bits per value are stored • Field values follow sequentially • Complex packing (intended to decrease message size) • Y x 10D = R + (X1 + X2) x 2E • R, E, D are as for simple packing • X1 is reference for group, X2 is scaled value in group (X1 removed) • Data are split into groups with similar values per group RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  28. Use of D and E D E Increment -1 0 10.000 -2 -3 12.500 0 4 16.000 -1 1 20.000 -2 -2 25.000 -3 -5 31.250 0 5 32.000 -1 2 40.000 -2 -1 50.000 -3 -4 62.500 -1 3 80.000 D E Increment 0 0 1.000 -1 -3 1.250 1 4 1.600 0 1 2.000 -1 -2 2.500 -2 -5 3.125 1 5 3.200 0 2 4.000 -1 -1 5.000 -2 -4 6.250 0 3 8.000 D  D - 1 Increment  Increment x 10 Here R = 0, X1 = 0, X2 = 1 Increment = Y = original value RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  29. R, E and D selection example Temperature on balcony: -15.0 to 35.0, ±0.2 • Set D to give increments in range 1 - 10 • D = 1, i.e scale by 10 • data are now: -150 to 350, ±2 • Set R to minimum • R = -150.0 (IEEE floating point) • data are now: 0 to 500, ±2 • Set E to give required precision • E = 1, i.e. data are in increments of 2 • Set bit width to give required range • Required range = scaled range / increment • Required range = 500 / 2 = 250 so 8 bits R = -150, E = 1, D = 1, bit width = 8 RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  30. Spatial differencing • Saves space for data which vary smoothly (order 1), or data whose variation varies smoothly (order 2), et c. • For order 1, data values, f, define new values, g: • g1 = f1, g2 = f2 - f1, …, gn = fn - fn-1 • 1, 2, 3, 5, 7, 8, 9, 11 becomes 1, 1, 1, 2, 2, 1, 1, 2 • For order 2, g values are replaced by h: • h1 = f1, h2 = f2, h3 = g3 - g2, …, hn = gn - gn-1 • 1, 2, 4, 10, 17, 27, 40, 57 becomes 1, 2, 1, 4, 1, 3, 3, 4 • Minimum value is subtracted to keep values positive Order 1 Complex packing 4 groups Order 2 Complex packing 3 groups RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  31. Section 6 Section 6: Bit map section • Octets 1 - 4: Length of section • Octet 5: Section number (6) • Octet 6: Bit map indicator (CT 6.0) • Octets 7 - xx: Bit map (if present) Code Table 6.0: Bit map section Code figure 0 1 - 253 254 255 Meaning Bit map is present and follows in this section Use bit map predefined by originating centre Re-use previously defined bit map Missing (no bit map present) RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  32. Bit map case study With bit map bit map (2000 x 2000 x 1) = 4,000,000 bits data ( x 1000 x 1000 x 8)  25,130,000 bits total  29,130,000 bits No data No bit map 2000 x 2000 x 8 = 32,000,000 bits 2000 2000 Data (8 bits per pixel) Conclusion: bit map may not always save space, but if more than two bits per data point (typically true) it usually will … consider cloud top height data, space corners, et c. RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  33. Updates not requiring code change • Type of changes concerned • Addition of code and or flag table entries • Agreement and implementation • “Fast Track” can be used • Table entries can be approved for pre-operational use within a few months of consideration and successful validation • Full formal CBS approval follows, available around 2 years later • Application updates • Encoder needs new code / flag table available • Encoder software needs updating to insert field • Decoder needs new code / flag table available • Down-stream application (beyond decoder) needs to be updated to process new field • Decoder software can remain unchanged RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  34. Updates requiring code change • Type of changes concerned • Changes to code structure (new edition), or new GDT, PDT or DRT • Agreement and implementation • ET/DRC considers a request and proposes update • Multilateral validation of proposed update • Full formal CBS approval follows, available around 2 to 4 years later • Application updates • Encoder needs new software and /or template available • Encoder software needs updating accordingly • Decoder needs new software and /or template available • Down-stream application (beyond decoder) needs to be updated to process new data • Long lead time necessary … consider PUMA example RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  35. Interfacing to third party software • Use of third party generic software simplifies implementation • NOAA provides GRIB2 software for free download (look at http://www.nws.noaa.gov/tdl/iwt) • ECMWF provides GRIBEX software - will be / is being updated for GRIB2 (look at http://www.ecmwf.int/products/data/software/grib2.html) • Typically software package is set of functions compiled into a library • User’s application sets up required parameters and calls function from library, which are referenced at run time • Specific (not generic) encoders and decoders are quite simple but less flexible RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  36. Some GRIB2 data from EUMETSAT • Cloud mask, CLM, every 15 minutes, ~3km resolution, via EUMETCast or archive request. • Cloud top height, CTH, every 60 minutes, ~9km resolution, via EUMETCast or archive request. • Cloud analysis image, CLAI, every 180 minutes, ~9km resolution, via EUMETCast or archive request. • Clear sky reflectance map, CRM, twice per week, ~3km resolution, via EUMETCast or archive request. • Multi-sensor precipitation estimate, MPE, every 30 minutes, ~5km resolution, via web site. • Fire detection, FIR, every 15 minutes, ~3km resolution, via anonymous FTP (evolving algorithm). RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  37. Case study A, precipitation data RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  38. Case study A, precipitation data • GRIB2 encoded file • Specific decoder program for GRIB2 precipitation data • Decoder output • Dump of GRIB2 file RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  39. Case study B, cloud mask data RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  40. Case study B, cloud mask data • GRIB2 encoded file • Original data to be encoded • Specific en/decoder program for EUMETSAT cloud mask data • Decoder output • Dump of GRIB2 file RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  41. Case study C, cloud analysis image data GRIB2 Cloud analysis image RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  42. Case study C, cloud analysis image data • GRIB2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Bit map used RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  43. Case study D, cloud top height data GRIB2 Cloud top height RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  44. Case study D, cloud top height data • GRIB2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Bit map used • Two repeats of Sections 4, 5, 6 and 7 RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

  45. Sources of GRIB2 information The information used in preparing this presentation is based on: Guide to WMO Table Driven Code Form Used for the Representation and Exchange of Regularly Spaced Data in Binary Form: FM 92-XII GRIB Written by Dr Cliff Dey & WMO Manual on Codes, WMO Publication No. 306, Vol. 1, Part B RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005

More Related