1 / 25

Data Interoperability

Data Interoperability. The HDF Group Staff. Interoperability. Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation. (From Wikipedia).

fletcher-moses
Download Presentation

Data Interoperability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Interoperability The HDF Group Staff HDF/HDF-EOS Workshop XIV

  2. Interoperability • Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation. (From Wikipedia) HDF/HDF-EOS Workshop XIV

  3. Background for data interoperability • Focus on Earth Sciences • Data needs to be shared across communities • Self-described data format • Many different types of data • Grids, moving-sensor multidimensional fields, time series, profiles, trajectories, geospatial framework data, points HDF/HDF-EOS Workshop XIV

  4. Why data interoperability? • Data format • Several data formats available HDF4/5, netCDF3/4, GRIB, BUFR, Binary • Data are created with some data formats by following different physical models • End users would like to use familiar tools to access data via different formats HDF/HDF-EOS Workshop XIV

  5. Why data interoperability? • Semantic/content interoperability • An example: the missing value of a physical variable • Tools or applications need to know the fill value of a physical variable • How can they identify? Normally via an attribute that stores the fill value • The name can be “Fill_value, _FillValue, badValue,……” HDF/HDF-EOS Workshop XIV

  6. Goals • Discuss a few cases about data interoperability • Hear opinions or issues about data interoperability from audience. HDF/HDF-EOS Workshop XIV

  7. Data Format Interoperability HDF/HDF-EOS Workshop XIV

  8. Case 1 • One set of APIs to handle multiple data formats HDF/HDF-EOS Workshop XIV

  9. Case 1 • netCDF4 • Combine powerful model and simplicity of netCDF with features of HDF5 • netCDF interface on top of HDF5 • CDM(Common Data Model) • Harmonize netCDF, HDF4, HDF5, OPeNDAP, GRIB and others HDF/HDF-EOS Workshop XIV

  10. Discussions • Any comments/sharings about the experiences with netCDF4 and CDM? HDF/HDF-EOS Workshop XIV

  11. Case 2 • Format conversions and translations HDF/HDF-EOS Workshop XIV

  12. Data format Conversions/Translations • Format conversions • The HDF4 to HDF5 conversion tool • The HDF-EOS2 to netCDF3/4 conversion tool • HDF-EOS to GeoTIFF conversion tool • …… • Data format Translations • netCDF tools to access HDF4/5 files via OPeNDAP • …… HDF/HDF-EOS Workshop XIV

  13. Use netCDF tools to access HDF via OPeNDAP Aqua/Aura Translation Layers Users HDF4/5 Handlers IDV libnc-dap AIRS/OMI DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4/5 Files HDF/HDF-EOS Workshop XIV

  14. netCDF4 to access HDF-EOS5 files • Augmentation • One file can be used for both EOS5 and NetCDF-4. • Note that EOS5 users are not affected at all. NetCDF-4 file Augmentation HDF-EOS5 file HDF-EOS5 Augmented HDF-EOS5 file NetCDF4 HDF5 HDF5 Sep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 14

  15. Discussions • Any comments/sharings about the experiences with data format conversions and translations HDF/HDF-EOS Workshop XIV

  16. Semantic/Content Interoperability HDF/HDF-EOS Workshop XIV

  17. An example • Degree of latitude and longitude • Champaign’s latitude is 40°6´38" N • It can be represented in two formats: • 400638 in DDDMMSS format • 40.1105556 in decimal format • How can the applications know which format the file is used? • Better to have a common standard to facilitate the exchange this kind of information HDF/HDF-EOS Workshop XIV

  18. Semantic/Content Interoperability • CF conventions become such a standard • Many applications/tools follow CF conventions to interpret the metadata HDF/HDF-EOS Workshop XIV

  19. Use netCDF tools to access HDF via OPeNDAP Aqua/Aura Not follow CF conventions Users HDF4/5 Handlers IDV libnc-dap AIRS/OMI DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4/5 Files HDF/HDF-EOS Workshop XIV

  20. Discussions • Any comments about semantic/content interoperability • Any comments about the usage of CF conventions HDF/HDF-EOS Workshop XIV

  21. HDF5 and netCDF-4 Libraries InteroperabilityTwo sources of interoperability problems and how to deal with them HDF/HDF-EOS Workshop XIV

  22. netCDF-4 library and arbitrary HDF5 files • netCDF-4 is built on top of HDF5 • netCDF-4 files are HDF5 files with specific characteristics • Tracked creation order of objects and their attributes • Absence of datasets with complex compound, and region and object references • Presence of dimension scales • netCDF-4 expects to find properties 1 and 2 in an HDF5 file and fails to open or read the file when those are not present (e.g., the NPOESS files) • Solution: TB discussed with the netCDF-4 folks HDF/HDF-EOS Workshop XIV

  23. Programming models and libraries settings • Use case 1 (libraries settings): Simultaneous access to an HDF5 file via HDF5 and netCDF-4 • One of the libraries will not be able to open the file due to the different access properties used by each library (H5F_CLOSE_WEAK in HDF5 vs. H5F_CLOSE_ STRONG in netCDF-4) • Solution: • Design improvements in HDF5 • New APIs to detect situation • Automatic detection and correction of the access properties • Document the best practices for applications HDF/HDF-EOS Workshop XIV

  24. Programming models and libraries settings • Use case 2 (programming models): Usage of HDF5 wrapper libraries and netCDF-4 library in the same application • C++ H5Topen call fails due to the netCDF-4 library shutting down the HDF5 library underneath by calling H5close • Solution: • Resolved in the newest netCDF-4 • Document the best practices for applications • For more detailed discussion see “NetCDF-4/HDF5 Libraries Interoperability Issues” at http://www.hdfgroup.uiuc.edu/RFC/HDF5/netCDF4-HDF5/ • Let us know your use case to improve testing! HDF/HDF-EOS Workshop XIV

  25. Thank you ! HDF/HDF-EOS Workshop XIV

More Related