Netcdf data model programming interfaces conventions and format
Download
1 / 23

NetCDF: Data Model, Programming Interfaces, Conventions and Format - PowerPoint PPT Presentation


  • 99 Views
  • Uploaded on

NetCDF: Data Model, Programming Interfaces, Conventions and Format. Adapted from Presentations by Russ Rew Unidata Program Center University Corporation for Atmospheric Research Sponsored by NSF and NASA. www.unidata.ucar.edu. Overview. What is Unidata? What is netCDF? Data model

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' NetCDF: Data Model, Programming Interfaces, Conventions and Format' - baba


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Netcdf data model programming interfaces conventions and format
NetCDF: Data Model, Programming Interfaces, Conventions and Format

Adapted from Presentations by Russ Rew

Unidata Program Center

University Corporation for Atmospheric ResearchSponsored by NSF and NASA

www.unidata.ucar.edu


Overview
Overview Format

  • What is Unidata?

  • What is netCDF?

  • Data model

  • Programming interfaces

  • Format

  • Limitations

  • Standardization Possibilities

  • Summary


Unidata a community endeavor
Unidata: A Community Endeavor Format

  • Community of educators and researchers at 120 universities, 30 other institutions, international in scope

  • Mission: providing data, tools, support, and community leadership for enhanced earth-system education and research

  • Funded primarily by the National Science Foundation

  • Atmospheric science community, expanding to oceanography, hydrology, other geosciences

  • Unidata Program Center: 25 staff, 15 developers


Unidata provides
Unidata Provides ... Format

  • User support, software development, maintenance, and community advocacy

  • Near real time “push” data dissemination: Internet Data Distribution among 170 institutions for use in research and education (IDD, LDM)

  • Tools for analysis and visualization (Gempak, IDV, McIDAS)

  • Data access and collections infrastructure (netCDF, udunits, THREDDS, ...)


Netcdf overview
NetCDF Overview Format

  • Model for scientific data: variables, dimensions, attributes, coordinates

  • Libraries for data access: C, Fortran, C++, Java, Perl, Python, Ruby, Matlab, IDL, ...

  • Associated Conventions convey semantics

  • Format for portable binary data that supports direct access, metadata, appending new data, shared access (note that users need not know anything about format)


Some applications for netcdf
Some Applications for netCDF Format

  • Gridded output from models: forecast, climate, ocean, atmospheric chemistry, ...

  • Observational data: surface, soundings, satellite, lightning, aircraft, radar, ...

  • Various scientific data archives

  • Chromatography, mass spectrometry, medical imaging, ...

  • In general: platform-independent, language-independent interface to insulate data providers from data consumers


Netcdf s niche
NetCDF’s Niche Format

  • Simple data model for scientific datasets

  • Portable, self-describing data

  • Appendable, sharable, archivable

  • Direct access for efficient subsetting

  • Metadata via attribute conventions such as CF

  • Flexible remote access via OPeNDAP, HTTP, WCS

  • Lots of applications: NCO, ncbrowse, ncview, IDV, IDL, MATLAB, ArcGIS, ...

  • Language interfaces include C, Java, Fortran, C++, Perl, Python, Ruby, ...


Netcdf data model
NetCDF Data Model Format

  • Variables: name, shape (list of Dimensions), type, Attributes, values

  • Dimensions: name, length

  • Attributes: name, type, value(s)

  • Coordinate Variables: values associated with Dimensions


Netcdf data model1

Dataset Format

Dimension

Attribute

name: String

length: int

isUnlimited: boolean

name: String

value: Vector

type: DataType

Variable

name: String

shape: Dimension[ ]

type: DataType

CoordinateVariable

ValuesArray

name: String

shape: Dimension[1]

type: DataType

type: primitive

shape: int[ ]

NetCDF Data Model

location: String

same name


Availability of netcdf
Availability of netCDF Format

  • Freely available source, binaries:

    • www.unidata.ucar.edu/packages/netcdf

  • Unidata-supported interfaces:

    • C, C++, Fortran-77, Fortran-90, Java, Perl

  • Platforms:

    • Unix, Windows, Mac OS X, ...

  • License:

    • X Windows/MIT style open source license

    • Used in several commercial applications

  • Status:

    • version 3.5.1 released February 2004

    • netCDF-4 under development


Widespread netcdf use
Widespread netCDF Use Format

  • 500 entries representing 32 in netcdfgroup mailing list: academic institutions, government agencies, commercial firms

  • Downloaded by more than 2000 distinct hosts in 55 countries in last 10 years

  • Software for manipulating/displaying netCDF: more than

    • 15 commercial packages,

    • 56 freely available package

  • Adopted by Analytical Instrument Association [AIA] to implement the Analytical Data Interchange Protocols

  • Neuro-Imaging Laboratory of the Montreal Neurological Institute


Netcdf language interfaces
netCDF Language Interfaces Format

  • Conventions of each language supported by interfaces

  • C: core implementation

  • C++, Fortran-77, Fortran-90, Perl: call C library

  • Java: independent pure-Java implementation

  • Python, Ruby, IDL, Matlab, Ada: third-party layers over C library

  • Data is language-neutral


Java interface
Java Interface Format

  • Currently most advanced netCDF interface, used to test new ideas

  • Client access to data servers:

    • HTTPD

    • OPeNDAP/DODS

  • Advanced coordinate system support for general and georeferenced coordinates

  • Virtual datasets to aggregate, subset, redefine, using NcML (XML representation for netCDF)


Netcdf format
NetCDF Format Format

  • Purpose: insulate users, applications, and data from machine architectures, format details, additions to data

  • Provides data that is

    • self-describing

    • portable

    • directly accessible

    • appendable

  • Still support original XDR-based format (1988)

  • 32-bit file offset limits file size

  • New format with 64-bit offsets for huge files


Netcdf file format

Records grow in the UNLIMITED dimension Format

NetCDF File Format

  • Header

    • Dimensions

    • Attributes

    • Variable metadata

  • Fixed-size variables

  • Record variables

    • Use UNLIMITED dimension

  • XDR-like represenations for all data and metadata


Some current netcdf limitations
Some Current netCDF Limitations Format

  • Not a database system: no keyed access, transactions, multiple concurrent writers

  • Strings are just character arrays

  • No nested data structures

  • No “ragged arrays”, variable length records

  • Only one growable dimension per dataset

  • No built-in compression or packing

  • Georeferencing requires agreement on conventions

  • ...


Netcdf cf conventions
NetCDF CF Conventions Format

  • Climate and Forecast metadata conventions

    • Metadata for use, not discovery

    • http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html

    • Adopted by: PCMDI, PRISM, ESMF, NCAR, Hadley Centre, GFDL, various EU projects

    • Developed by volunteers, active email list

  • Includes georeferencing conventions for multidimensional coordinate systems

  • Includes a “Standard Name Table” for quantities

    • Google for “CF standard name table”

    • 76 of 416 entries (as of June 3, 2004) relate to hydrology


Taking advantage of web services for data system interoperability

FES Client Format Applications

GIS Client Applications

FES Servers

GIS Servers

THREDDS Server

GIS Server

THREDDS Server

OGC or OPeNDAPADDE. FTP…

protocols

OpenGIS Protocols:

WMS, WFS, WCS

Satellite, radar, forecast model output, … datasets

OGC or proprietary GISprotocols

GIS Server

Demographic, infrastructure, societal impacts, … datasets

Taking Advantage of Web Services for Data System Interoperability


Obstacles to gis netcdf connection
Obstacles to GIS-netCDF Connection Format

  • netCDF data model (finite sampling of real functions) differs from relational, GIS datamodels

  • Some netCDF data not spatial

  • Much netCDF data multidimensional, > 2D

  • Georeferencing for netCDF data uses multiple conventions

  • Typical netCDF datasets may be very large

  • Large netCDF datasets may be on a remote server


Unidata gis netcdf efforts
Unidata GIS-netCDF Efforts Format

  • Prototype netCDF to geoTIFF converters

  • Parser for georeferencing conventions implemented in Java

  • Unidata GeoGrid from georeferenced netCDF variable supported in Java

  • GridDataset: collection of GeoGrids in a netCDF dataset with methods for getting volumes and slices

  • NcML for wrapping any new conventions around existing data


Netcdf related changes proposed in wcs 1 1
netCDF-related FormatChanges Proposed in WCS 1.1

  • WCS encoding “profiles” instead of fixed list of encoding formats

  • CF-netCDF is first proposed WCS encoding profile

  • Multiple “variables” or “parameters” in a coverage (e.g., pressure, temperature, etc.)

  • Coverages with 3 spatial dimensions

  • Coverages with multiple time dimensions (e.g. forecast time in model output)

  • Non-spatial “height” dimension, (e.g., atmospheric pressure, ocean density)

  • Irregularly-spaced grids

  • Technical committee votes on entirety of WCS 1.1 this summer.


Possible netcdf standardization
Possible netCDF Standardization Format

  • File format

  • Applications programming interfaces

  • Conventions (e.g., CF, Climate and Forecast)

  • Release level:

    • netCDF 3

    • netCDF 4 later this year (will include access to HDF 5 files)

  • Common Data Access Model (with elements of netCDF, HDF, and OPeNDAP data models)

  • (Relationship to WCS encoding profile specification?)


More information
More Information Format

  • NetCDF documentation and codehttp://www.unidata.ucar.edu/software/netcdf/

  • CF Conventionshttp://www.cgd.ucar.edu/cms/eaton/cf-metadata/index.html

  • Russ [email protected]


ad