grib2 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Grib2 PowerPoint Presentation
Download Presentation
Grib2

Loading in 2 Seconds...

play fullscreen
1 / 60

Grib2 - PowerPoint PPT Presentation


  • 1451 Views
  • Uploaded on

Wesley Ebisuzaki (wesley.ebisuzaki@noaa.gov) wgrib2: Wesley Ebisuzaki, Reinoud Bokhorst, Jaakko Hyvätti, Kristian Nilssen, Karl Pfeiffer, Pablo Romero, Manfred Schwarb, Arlindo da Silva, Niklas Sondell, Sergey Varlamov. Grib2. what is grib2, new features GrADS & grib2

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Grib2' - kiet


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
grib2

Wesley Ebisuzaki (wesley.ebisuzaki@noaa.gov)

wgrib2: Wesley Ebisuzaki, Reinoud Bokhorst, Jaakko Hyvätti, Kristian Nilssen, Karl Pfeiffer, Pablo Romero, Manfred Schwarb, Arlindo da Silva, Niklas Sondell, Sergey Varlamov

Grib2
outline

what is grib2, new features

GrADS & grib2

converting from wgrib to wgrib2

inventory grib2 files

geolocation

read grib2 with wgrib2

write grib2 with wgrib2

Outline
what is grib2
What is grib2
  • WMO replacement for grib version 1
  • Transfer of gridded data between met centers
  • Software to read/write grib2
    • NCO fortran/C libraries, used by many codes
    • NCAR library, NCL (NCAR command language)
    • ECMWF grib api, used by many codes
    • wgrib2
    • others: UK? Japan?
why grib1 grib2
Why grib1 -> grib2
  • Address the limitations of grib1
  • Time stamp
    • Only two bytes to store numbers (255 is ok)
    • can combine 2 bytes for 16 bit integer (64k is ok)
  • Files size limitation
  • Only 128 standard variables, all other are locally defined
  • Grids only defined to 1/1000 of a degree
new features in grib2
New features in grib2
  • Submessages
  • Improved metadata: time stamps, level information, variable information
  • Default grid resolution is 1e-6 degrees but can be specified to be higher
  • *** Selectable compression schemes ***
gone kpds 5 kpds 7
Gone: kpds(5) .. kpds(7)
  • Grib1: variable, type of level, level #
  • Grib2:
    • variable: table/version, discipline, parameter category, parameter number
    • New tables to modify variables, ensemble, bias, error, chemical type, statistical processing, etc
    • level: most used templates require several #s.
  • wgrib2, GrADS: hide complexity, similar to grib1
forecast timing information
Forecast timing information
  • Grib1: 4 bytes: type, time units, 2 bytes
  • Grib2: size of time stamp is not fixed
    • No overflows, precise descriptions
    • Means are handled is systematic manner
      • in wgrib2 notation, monthly means look like
      • “124@6 hour ave(6 hour fcst)”
      • “124@6 hour ave(0-6 hour ave fcst)”
    • Statistical operations: ave, acc, min, max, rms, etc
submessages

bash-2.05a$ wgrib2 wafsgfs44.t18z.gribf84.grib2

1:0:d=2011011218:HGT:70 mb:84 hour fcst:

2:5409:d=2011011218:TMP:70 mb:84 hour fcst:

3.1:8037:d=2011011218:UGRD:70 mb:84 hour fcst:

3.2:8037:d=2011011218:VGRD:70 mb:84 hour fcst:

4:16727:d=2011011218:HGT:100 mb:84 hour fcst:

5:22349:d=2011011218:TMP:100 mb:84 hour fcst:

Note: 3.1 and 3.2, UGRD and VGRD are in same grib message, share grid description and metadata.

GrADS, wgrib2: process by “fields” not by messages.

Fortran: extra complexity

Submessages are good for irregular grids. Only need to store one copy of the latitude/longitude values.

Submessages
bitmaps undefined values
Bitmaps & Undefined Values
  • Grib1: uses bitmap for undefined grid value
    • overhead = 1 bit per grid point
  • Grib2: use bitmap or special value(s)
    • All packing methods can use bitmap
    • Complex packing can use special values
    • better to use special values if available
  • GrADS, wgrib2 hides this complexity
  • fortran code: extra complexity
scan order
Scan order
  • Decode a grib field – get a list of grid pt values
  • “scan order” specifies how you fill the grid
  • Grib1: we:sn and we:ns are common
  • Grib2: 16 different orders but 3 are common, we:sn, we:ns and “Glahn” (NDFD)
  • wgrib2: default is to convert data to we:sn but other options include we:ns and raw but best to keep in we:sn order
grib1 grib2 storing grid data
Grib1, Grib2 – storing grid data
  • Store scaled integers
  • Value = (i * 2^E + ref) * 10^D
  • ref = min value of the field so i >= 0
  • E, D are scaling factors for the field
  • Global model: E = 0, D is specified
  • Mesoscale: E and D are specified
  • ECMWF/wgrib2(default): D = 0, E determined by # bits wanted
packing schemes ieee
Packing schemes: IEEE
  • IEEE
    • store as IEEE real numbers
    • Allows full precision files (ex. restart files)
    • No compression
    • Easy to write (no grib library needed)
    • Not supported by NCO software
simple packing
Simple packing
  • Similar to grib1 packing
    • Store scaled (positive) integers
    • If biggest scaled integer < 2**N
    • Store integer as N bits
    • No compression relative to grib1
    • Fast to write and read
    • common
slide14
PNG
  • Compress file by “PNG” compression
    • same png compression as .png files
  • Better packings are available
  • Not recommended
slide15
JPEG
  • Uses JPEG2000 to compress the grid data
  • Best compression for data without undefineds
  • Slow to write and read
  • Used by NCO
complex packing
Complex Packing
  • Start with scaled integer, 3 variants
    • C1: store values (Vi)
    • C2: store increments (Vi - Vi-1)
    • C3: store increments of increments
  • Split into multiple regions
    • each region stores packed integer with varying minimum value and size
complex packing17
Complex Packing
  • Heuristic methods to determine regions
  • Some programs pack more efficiently
  • Best to use special values rather than bitmap
    • Undefined encoded as a run of N “special values”
  • Reading complex packed files is fast. Similar to grib1/simple packing.
complex packing file size
Complex Packing: file size
  • C1/C2/C3 is best for noisy/somewhat noisy/smooth fields
  • Can produce better complex packing by making C1,C2,C3 and choosing the best (C0)
  • C0 produces smaller files than JPEG for files with undefined values
  • C0 typically 10%-20% bigger than jpeg
examples file size
Examples: file size
  • 1/14/2011 gfs.t00z.sfluxgrbf06.grib2
      • grib1: 213 MB jpeg: 65.1 MB C0: 65.9 MB * 50.7 MB
  • 1/14/2001 gfs.t00z.prgb2f06
      • jpeg 55.1 MB C0: 56.5 MB * 36.0 MB
  • 1/14/2011 nam.t00z.edasf06.tm
      • grib1: 114 MB jpeg: 38.1 MB C0: 34.1 * 24.8 MB
  • * use best of jpeg, C1, C2, C3
compression summary
Compression: summary
  • JPEG slow to write and read
  • Complex packing is fast to read
  • Complex packing is smaller for files with undefineds otherwise usually 10-20% bigger
  • Tapes: best compression (jpeg+C0)
  • Speed: C0
  • Multitasking routines: jpeg <-> C0..C3
grib2 common tasks
Grib2: common tasks
  • Display with GrADS
  • Inventory
  • Slicing and dicing files
  • Exporting data
  • Averaging
  • Making regional subsets
  • Making grib2 content: conversion, new data
grib2 grads
Grib2 & GrADS
  • Use GrADS v2, g2ctl
  • Supports ensemble
  • Supports minutes
  • Supports 3 time mapping schemes
    • -0, -b, -verf (use “”)
grads time dimension

Mapping grib time to GrADS time dimension.

:d=2009060500:APCP:surface:324-330 hour acc fcst:

d=2009060500 analysis time (-0)

d=2009060500+324 hours start of verf period (-b)

d=2009060500+330 hours end of verf period (“”)

d=2009060500:UGRD:1000 mb:330 hour fcst

d=2009060500 analysis time (-0)

d=2009060500+330 hours verf time (-b or “”)

GrADS: time dimension
reference analysis time use 0 g2ctl 0 ipvhnl gdas y4 m2 d2 h2 grb2 anl ctl gribmap 0 i anl ctl
Reference, analysis time: Use -0

g2ctl -0 ipvhnl.gdas.%y4%m2%d2%h2.grb2 >anl.ctl

gribmap -0 -i anl.ctl

GrADS examples
grads examples

start of verification time period: Use -b option

g2ctl -b gfs.t12z.pgrb2.f%f3 >fcst.ctl

gribmap -b -i fcst.ctl must be gribmap v2

end of verification time period: No extra options

g2ctl gfs.t12z.pgrb2.f%f3 >fcst.ctl

gribmap -i fcst.ctl must be gribmap v2

GrADS examples
grads examples26

(forecast) ensemble:

files: gep01.t00z.pgrb2cf*

gep02.t00z.pgrb2cf*

gep03.t00z.pgrb2cf*

g2ctl -ens “01,02,03” gep%e.t00z.pgrb2cf%f3 >ens.ctl

gribmap -i ens.ctl

Can use -b in the above example.

GrADS examples
introduction to wgrib2
Introduction to wgrib2
  • wgrib2 was planned
    • Easy to add modules
    • People donated modules
      • Geolocation
      • Name <-> parameter routines
      • Update of tables from NCO web pages
      • mysql export
      • netcdf export
wgrib2 changes
wgrib2 changes
  • Each option corresponds to a subroutine call
    • Order is important
    • command line can look like a small program
  • Many options, to view
    • wgrib2 will list main options
    • wgrib2 -help all will list all options
  • Functionality
    • Geolocation, if, for, conversion to other formats, writing grib2, interpolation, multitasking
inventory
Inventory
  • Simple inventory:
    • wgrib2 -s GRB_FILE
    • wgrib2 GRB_FILE -s is assumed, no need -4yr
  • Verbose inventory
    • wgrib2 -V GRB_FILE
  • -s and -V are inventory options
inventory30
Inventory
  • Inventory has changed
    • old names are mostly the same
    • new levels, old levels are expanded
    • years are 4 digits long
    • time stamps can be more complicated
    • NO KPDS5, KPDS6, KPDS7 or equiv g2lib info
slicing and dicing
Slicing and Dicing
  • Grib1: wgrib IN -s -4yr | egrep (...) | wgrib IN -i -grib -o OUT
  • Grib2: wgrib2 IN | egrep (...) | wgrib2 IN -i -grib OUT
  • -4yr, -s are gone, now 4 digit year codes
  • Notice: -grib -o OUT becomes -grib OUT
  • Minor changes to scripts but will need to debug
  • Watch out for scripts that write binary/ieee/text files
    • default is we:sn rather than raw. (we:ns in global)
efficiency match not
Efficiency: -match, -not
  • wgrib2 IN | egrep A | egrep -v B | wgrib2 -i IN ...
  • A, B are regular expressions
  • 4 processes, read file up to 2x
  • wgrib2 IN -match A -not B ...
  • multiple -match and -not options
  • More efficient: 1 read and 1 process
efficiency multiple write
Efficiency: multiple write
  • wgrib2 IN -match ':TMP:' -grib ht.grb
  • wgrib2 IN -match ':HGT:' -append -grib ht.grb
  • wgrib2 IN -match ':RH:' -grib rh.grb
  • OR with one command line
    • wgrib2 IN -if ':TMP:' -grib ht.grb \ -if ':HGT:' -grib ht.grb \ -if ':RH:' -grib rh.grb
  • -if is ended by i/o operation or -fi
geolocation

A need to know the values at specified lat-lon.

A need to know the lat-lon of the grid points.

-lon LON LAT grid value nearest to lat-lon

-ijlat I J values and lat-lon of grid point (i,j)

-lola ARGS makes lola file with NN interpolation

-csv FILE CSV with lat-lon values

-spread FILE CSV with lat-lon values

Geolocation
wgrib2 exporting data
wgrib2: exporting data
  • Many output formats
    • ieee -ieee OUTPUT.bin big endian IEEE
    • bin -bin OUTPUT.bin machine (IEEE)
    • text -text OUTPUT.txt
    • csv -spread OUTPUT.txtcomma sep. values
    • csv -csv OUTPUT.txt another CSV
    • mysql -mysql ... to mysql
    • netcdf -netcdf ... write netcdf file
exporting to fortran
Exporting to fortran
  • Use -bin OUT.bin(binary)
    • Default is -header
    • open(...,form='unformatted',access='sequential')
    • If you use -no_header option, get direct access file.
    • open(..,recl=nx*ny*4,form='unformatted',access='direct')
  • Scan order is we:sn by default, use -order raw for original order
exporting to fortran37
Exporting to fortran
  • Order and number of the records can change!
  • Be prepared for potential changes
  • Specify the records that you want (and count)
  • Sort the inventory before decoding to order
  • wgrib2 IN | egrep “:(UGRD|VGRD):(2|3)00 mb” | sort -t: -k3,3 -k4,4 -k5,5 -k6,6 | wgrib2 IN -i -ieee sorted.bin
only 1 record to read

call system('grep ":DPT:surface:60 hour fcst:" ds.td.bin.inv |

wgrib2 -i ds.td.bin -bin fort.20',status)

if (status /= 0) stop 8

open(unit=20,file='fort.20',form='unformatted',access='sequential')

read(20) rdata

close(20)

Only 1 record to read?
averaging

Use wgrib2: -ave dt GRIB.OUT

dt = N(time unit), ex 6hr, 1dy

The records have to be in sequence

d=2011010100:UGRD:1000 mb:anl:

d=2011010106:UGRD:1000 mb:anl:

d=2011010112:UGRD:1000 mb:anl:

d=2011010118:UGRD:1000 mb:anl:

wgrib2 IN | sort -t: -k4,4 -k5,5 -k6,6 -k3,3 | wgrib2 -i IN -set_grib_type c3 -ave 6hr AVE_OUT

Averaging
regional subsets
Regional Subsets
  • -small_grib lonE:lonW latS:latN OUT
    • Cookie cutter of existing grid
  • -ijsmall_grib i1:i2 j1:j2 OUT
    • Cookie cutter of existing grid (i,j) coordinates
  • -new_grid A B C OUT
    • Uses IPOLATES like copygb
    • A, B, C are new grid definition
new grid
-new_grid
  • subset of copygb
  • calls NCEP iplib library
  • not all functionality of iplib is supported
  • -new_grid is optional, not in opn version
using new grid

Type of interpolation

-new_grid_interpolation TYPE bilinear, neighbor, budget etc

Interpolation parameters

-new_grid_iopts I:J:K iopts(i) array

Type of winds (a must)

-new_grid_winds TYPE earth, grid

Notes: bilinear interpolation is default

Must set the type of winds (earth/grid)

using -new_grid
new grid example

-new_grid latlon lon0:nlon:dlon lat0:nlat:dlat OUT.grb

wgrib2 IN -new_grid_winds earth -new_grid latlon 0:360:1

90:181:-1 OUT

Some NCEP grids are built-in:

wgrib2 IN -new_grid_winds -new_grid ncep grid 3 OUT

input projections: latlon, mercator, polar, lambert, gaussian

output projections: latlon, gaussian, lambert, nps, sps

-new_grid: example
why a calculator
Why a calculator?
  • CSV: change units from Kelvin to Celcius
  • Net flux = downward flux – upward flux
  • wind speed = sqrt(u*u + v*v)
  • (3-6 hour prate) = 2*(0-6 hour prate) – (0-3 hour prate)
  • growing degree days
  • 500-1000 mb thickness = hgt(500mb) – hgt(1000mb)
rpn calculator46
RPN calculator
  • old fashioned, stack based calculator
  • 10 registers 0..9, stack is 10 deep
  • works on grids
  • operators +,-,*,/,pop,exc,trig,sqrt,log,exp,etc
  • ex. convert from K to C: -rpn “273.15:+”
saving calculator results

The calculation replaces the grid data read from the grib file. The new data can be written out with the standard wgrib2 output operators except for grib data, you have to use -grib_out.

Trivial example: changing sign of LHTFL

wgrib2 -match “:LHTFL:” -rpn “-” -grib_out neg_lhtfl.grb

Saving calculator results
calculating wind speed

wgrib2 IN.grb -match GRD \

-if “:UGRD:500 mb:” -rpn sto_1 -fi \

-if “:VGRD:500 mb:” -rpn sto_2 -fi \

-if_reg 1:2 -rpn “rcl_1:sq:rcl_2:sq:+:sqrt:clr_1:” \

-set_var WIND -grib_type jpeg \

-grib_out out.grb

calculating wind speed
changing metadata
Changing metadata
  • previous slide: -set_var WIND
  • many options to alter the metadata
  • -set_var HGT
  • -set_date 2011010212
  • -set_ftime “12 hour fcst”
  • -set_lev “200 mb”
  • -set_metadata FILE
writing out new grib2 file
Writing out new grib2 file
  • Select compression
  • -set_grib_type (jpeg|s|c1|c2|c3)
  • Compress data and write new grib message
  • -grib_out NEW.grb
  • Examples of combining -rpn, -set_* and -grib_out to write a new grib message are on web site
writing grib import bin
Writing grib: -import_bin
  • -import_bin allows over write the grid data
    • wgrib2 T.GRB -import_bin NEW.bin -grib_out NEW.grb
    • -import_bin, reads a binary file and overwrite data
    • -grib_out writes out a new grib2 file
    • useless example – same variable/time/level
    • change metadata by: -set_date, -set_var, -set_lev, -set_ftime, etc
conversion scripts
Conversion Scripts
  • GrADS can read many file formats such as binary, grib, netcdf, hdf
  • If GrADS can read the file, can convert the data into grib2 by the g2grb.gs script
  • Used in converting MERRA to grib2
  • Limitations: only lat-lon grids supported, Lambert/polar converted to lat-lon grid
summary
Summary
  • Grib2 is a more complicated format than grib1
    • Compression
    • Fixes limitations of grib1
    • Some grib2 fields have no grib1 versions
  • Conversion from wgrib to wgrib2 is straight forward
  • Take advantage of the new features in wgrib2
  • export data to csv, netcdf, mysql
summary54
Summary:
  • can use wgrib2 to write grib2 files
    • -rpn calculator
    • -import_bin import binary data
    • use -set options to modify metadata
    • -grib_out to write grib files
  • multi-processing can be done (see web site)
rle run length encoding
RLE: run length encoding
  • store as N < 255 discrete IEEE values
  • radar data (Japan)
  • Works well in this application
  • Limited use
grib1 conversion scripts
Grib1 Conversion Scripts
  • Everybody has a grib1->grib2 program
    • cnvgrib, ECMWF api, grib1to2.pl
  • Grib1 -> grib2 (used for CFSR conversion)
    • wgrib -> ieee file
    • wgrib + perl -> metadata file
    • cnvgrib -> grib2 template
    • wgrib2 + ieee + metadata + template = grib2 file
    • 228 lines (1 cpu version), 278 lines (multi-cpu)
parallel operations
Parallel operations
  • wgrib2 can be used for cpu intensive operations such as compression and regional subsetting.
  • On the CCS, a 30 sec job only took 8 seconds when run on 4 copies of wgrib2
  • -for_n I:J:K option uses do-loop syntax to select the fields to process.
  • -for_n/-for can also be used to select blocks to process -for 10:20 will process msgs 10-20
example multicpu jpeg
Example: multicpu -> jpeg
  • # 1-cpu: wgrib2 IN -set_grib_type jpeg -grib_out OUT
  • mkfifo pipe1 pipe2 pipe3
  • wgrib2 IN -for_n 1::3 -set_grib_type jpeg -grib_out pipe1 &
  • wgrib2 IN -for_n 2::3 -set_grib_type jpeg -grib_out pipe2 &
  • wgrib2 IN -for_n 3::3 -set_grib_type jpeg -grib_out pipe3 &
  • gmerge OUT pipe1 pipe2 pipe3
  • IN = input grib file, OUT = output grib file