1 / 18

Advanced Utilities Extending ncgen to support the netCDF-4 Data Model

Advanced Utilities Extending ncgen to support the netCDF-4 Data Model. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009. Overview. The NCGEN4 Utility NCGEN4 Command Synopsis netCDF4 in CDL: Types netCDF4 in CDL: Typed Attributes netCDF4 in CDL: Groups Scope Rules

fleta
Download Presentation

Advanced Utilities Extending ncgen to support the netCDF-4 Data Model

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Advanced UtilitiesExtending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009

  2. Overview • The NCGEN4 Utility • NCGEN4 Command Synopsis • netCDF4 in CDL: Types • netCDF4 in CDL: Typed Attributes • netCDF4 in CDL: Groups • Scope Rules • Specifying Data Constants • Rules for Using Braces • Special Attributes • Debugging Note: The Cycle • Building and Installing NCGEN4 • Extended Example

  3. The NCGEN4 Utility • NCGEN4 extends the CDL language • Includes all of the netCDF-4 data model • True inverse of ncdump • Includes the special attributes (chunking, etc) • Supports binary, C, NOT FORTRAN (yet) • Experimental: Java and NcML • Can produce 4 kinds of binary netCDF files: • netcdf-3 (classic) 32 bit • netcdf-3 (classic) 64 bit • netcdf-3 (classic, but stored in netcdf-4 file format) • netcdf-4 (supports full netcdf-4 model)

  4. NCGEN4 Command Synopsis Primary Command Line Options [-b] Create a (binary) netCDF file (default) [-o <file>] Name for the binary netCDF file created [-k <file_format>] Format of the file to be created 1 => classic 32 bit 2 => classic 64 bit 3 => netcdf-4/CDM 4 => classic, but stored in an enhanced file format [-x] Don't initialize data with fill values [-l <language>] Specify output language to use when generating source code to create or define a netCDF file matching the CDL specification. <filename> Input CDL file

  5. netCDF4 in CDL: Types • New section called “types:” • Consistent with the output of ncdump • Supports the new primitive data types: ubyte, ushort, uint, string, int64 (LL), uint64 (ULL) • Supports the new user-defined types: • int enum enum_t {off=0,on=1,unknown=2}; • opaque(11) opaque_t; • compound cmpd_t { vlen_t f1; enum_t f2;}; • int(*) vlen_t;

  6. netCDF4 in CDL: Typed Attributes • vlen_t v:attr = {17, 18, 19}; • Attribute typing is optional (=> type inferred) • Warning! x:attr = “abc”; is inferred to be type char, not string • Instead say string x:attr = “abc”; • Why? for backward compatibility with ncgen • Good practice to add “_t” to the end of all type names • Why? Because X :attr = … might be interpreted incorrectly; is X a type or variable?

  7. netCDF4 in CDL: Groups • group: g {…} • A group can itself contain dimensions, types, variables, and groups • Name prefixing allows references to types and dimensions that reside in other groups • Example: /g/cmpd_t => Do not use ‘/’ in your names • Pretty much like the Unix file system • Or Windows, but using forward slashes

  8. Scope Rules • Scope rules determine how references to a dimension or type without a prefix are interpreted • General rule: • Look in immediately enclosing group • Look in the parent of the immediately enclosing group and so on up the enclosing groups • For dimensions, if not found => error • For types, continue to search the whole group tree to find a unique match, then error if not found

  9. Specifying Data Constants • Constants for user defined types require the use of braces {…} in certain places. dimensions: d=2; types: int(*) vlen_t; compound cmpd_t { int64 f1; string f2;}; variables: vlen_t v1(d); cmpd_t v2(d); data: v1 = {7, 8, 9}, {17,18,19}; v2 = {107LL, “abc”}, {1234567LL, “xyz”};

  10. Rules for Using Braces • The top level is automatically assumed to be a list of items, so it should not be inside {...} • Different than C constants lists • Instances of UNLIMITED dimensions (other than as the first dimension) must be surrounded by {...} in order to specify the size. • Instances of VLENs must be surrounded by {...} in order to specify the size. • Compound instances must be embedded in {...} • Compound fields may optionally be embedded in {...}. • No other use of braces is allowed.

  11. Special Attributes • Special attributes specified in an ncgen4 CDL file will be properly handled • Consistent with ncdump -s • Global special attributes • “_Format” – specify the netCDF file format • “classic” • “64-bit offset” • “netCDF-4” • “netCDF-4 classic model” • Overridden by the -k flag

  12. Special Attributes (cont.) • Per-variable special attributes • “_ChunkSizes” – list of chunk sizes 1 per dimension • “_DeflateLevel” – compression level: integer (0-9) • “_Endianness” – “big” or “little” • “_Fletcher32” – “true” or “false” to set check summing • “_NoFill” – “true” or “false” to set persistent NoFill property • “_Shuffle” – “true” or “false” to set shuffle filter • “_Storage” – “contiguous” or “chunked” to set storage mode

  13. Debugging Note • Use the “Cycle”, Luke • Use ncgen/ncgen4 to convert your <file>.cdl to <file>.nc • Then use ncdump to convert your <file>.nc to <file2>.cdl • Compare <file>.cdl to <file2>.cdl • Watch out for UNLIMITED! dimensions: u = unlimited; variables: v1(u); v2(u); data: v1 = {1,2,3,4}; v2 = {7,8}; Ncdump produces v2 = {7,8,_,_};

  14. Building and Installing NCGEN4 • Easy: add --enable-ncgen4 to your list of ./configure flags • Ncgen4 will be installed along with ncdump and the original ncgen

  15. Extended Example netcdf foo { types: ubyte enum enum_t {Clear = 0, Cumulonimbus = 1, Stratus = 2}; opaque(11) opaque_t; int(*) vlen_t; dimensions: lat = 10; lon = 5; time = unlimited ; variables: long lat(lat), lon(lon), time(time); float Z(time,lat,lon), t(time,lat,lon); double p(time,lat,lon); long rh(time,lat,lon); string country(time,lat,lon); ubyte tag;

  16. Extended Example (cont.) // variable attributes lat:long_name = "latitude"; lat:units = "degrees_north"; lon:long_name = "longitude"; lon:units = "degrees_east"; time:units = "seconds since 1992-1-1 00:00:00"; // typed variable attributes string Z:units = "geopotential meters"; float Z:valid_range = 0., 5000.; double p:_FillValue = -9999.; long rh:_FillValue = -1; vlen_t :globalatt = {17, 18, 19};

  17. Extended Example (cont.) data: lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90; lon = -140, -118, -96, -84, -52; group g { types: compound cmpd_t { vlen_t f1; enum_t f2;}; } // group g group h { variables: /g/cmpd_t compoundvar; data: compoundvar = { {3,4,5}, Stratus } ; } // group h }

  18. Questions?

More Related