Hdf5 opendap project update and demo
This presentation is the property of its rightful owner.
Sponsored Links
1 / 26

HDF5 OPeNDAP Project Update and Demo PowerPoint PPT Presentation


  • 53 Views
  • Uploaded on
  • Presentation posted in: General

HDF5 OPeNDAP Project Update and Demo. MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.). OPeNDAP Review. A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community.

Download Presentation

HDF5 OPeNDAP Project Update and Demo

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Hdf5 opendap project update and demo

HDF5 OPeNDAP ProjectUpdate and Demo

MuQun Yang and Hyo-Kyung Lee

(The HDF Group)

James Gallagher

(OPeNDAP, Inc.)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

OPeNDAP Review

  • A software framework that allows

  • simple access to remote data

  • Data Access Protocol

  • Client-server model

  • Widely used by Earth Science

  • Community

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

OPeNDAP Concept

Typical Remote Access

View Data

(User)

Remote Data

(HDF4)

FTP/HTTP

Visualization Tools

(gradsdap)

Visualization tools

(e.g. grads)

Handler

(hdf4_handler)

Client Library

(libnc-dap)

Server

(apache)

DAP Protocol (via http)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

OPeNDAP HDF5 Handler

View Data

(User)

Remote HDF5

HDF5 Handler

Server

(apache)

DAP Protocol (via http)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Challenges

Mapping HDF5 to DAP

  • Compound Datatype

  • Groups

  • Object/Regional References

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

OPeNDAP HDF5 Handler with HDF-EOS5

View Data

(User)

Remote

HDF5

Remote

HDF-EOS5

Visualization tools

(grads)

HDF5 Handler

Client Library

(libnc-dap)

Server

(apache)

DAP Protocol (via http)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Challenges: HDF- EOS5

  • Grid with No Geolocation data

    • Clients expect Grid with Geolocation data

  • Some Attributes stored as Extremely Long

  • String(s) (e.g. StructMetada.0)

    • Clients expect structured attributes

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Tweaks for HDF-EOS5

  • Added two default HDF5 handler configuration options:

  • --enable-eos-grid

  • --enable-eos-meta

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Challenges: OPeNDAP Clients

  • Need special attributes on dataset.

  • Need shared geolocation variables.

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Tweaks for OPeNDAP Clients

  • Added two optional handler configuration options:

  • --enable-short-name

  • --enable-CF

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

OPeNDAP HDF5 Handler with HDF-EOS5

View Data

(User)

Remote

HDF5

Remote

HDF-EOS5

Visualization tools

(grads)

HDF5 Handler

HDF5 Handler

w/ CF options

Client Library

(libnc-dap)

Server

(apache)

DAP Protocol (via http)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Day After Server Tweaks

  • Finally, Happy Clients!

IDL

ncBrowse

Ferret

MATLAB

NCL

IDV

GrADS

ODC

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Problems of Tweaking HDF5 Handler

Remember that we added two optionalHDF5 handler configuration options:

--enable-short-name

--enable-CF

Causes ambiguity among

variable names

(e.g. /GroupA/ozone vs.

/GroupB/ozone)

Drops some key attributes

(e.g. StructMetdata, HDF_ROOT_GROUP)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

  • HDF5-Friendly OPeNDAP Client Library

View Groups

???

HDF5 Groups

View Data

(User)

Remote

HDF5

Visualization tools

(grads)

HDF5 Handler

HDF5 Handler

w/ CF options

Client Library

(libnc-dap)

HDF5-Friendly

Client Library

(liboc-dap)

Server

(apache)

DAP Protocol (via http)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.

Example: Groups in HDF5

HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Example: Reference in HDF5

  • Important for NPOESS

  • Object / Regional Reference

  • Map to DAP URL

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Example: dap2h5

View Group/Ref.

Group/Ref.

View Data

(User)

Remote

HDF5

  • A test application for the client library

  • It can construct HDF5 from DAP output

HDF5-F. Client Library App.

(dap2h5)

HDF5 Handler

ORIGINAL

CONSTRUCTED

HDF5-Friendly

Client Library

(liboc-dap)

Server

(apache)

DAP Protocol (via http)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Our Client Library(prototype)

  • One more reason:

  • Help Clients to view Swath

No Latitude and Longitude

Courtesy of NASA

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Visualizing HDF-EOS5 Swath Problem

Visualizing HDF-EOS5 Grids

View Swath

???

Swath

View Data

(User)

Remote

HDF-EOS5

Remote

HDF5

Visualization tools

(grads)

Visualization Tools

(gradsoc)

HDF5 Handler

HDF5 Handler

w/ CF option

HDF5-Friendly

Client Library

(liboc-dap)

Client Library

(libnc-dap)

Server

(apache)

DAP Protocol (via http)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Demo: MLS swath via GrADS

  • GrADS coupled with our client

  • library

  • The client library provides grid

  • mapping from swath data

  • GrADS displays swath through

  • HDF-EOS5 specific client library

  • API calls

HDF and HDF-EOS Workshop XII


Summary of client library prototype

Summary of Client Library (Prototype)

  • Finished coding

  • Tested with the GrADS visualization client

  • Documented

  • Working on a demo DAP to HDF5 tool by using the client library prototype

  • Will test with NCL if time allows

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Caution

  • Our Client Library is ONLY A PROTOTYPE!

    • It does NOT support all DAP data types

    • It does NOT support all AURA files

    • It does NOT support all Visualization clients

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Summary

  • HDF5 Access via OPeNDAP

    • Is Easy

    • Is used by GES DISC to serve Aura

    • files

  • HDF5-Friendly OPeNDAP Client Library

    • Serves HDF5 better(EOS swath)

    • Benefits visualization clients

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Future Work

  • HDF5 to DAP2 Mapping Document

  • Release HDF5-friendly OPeNDAP

  • Client Library (Prototype)

  • URL: http://hdfgroup.org/projects/opendap

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Credits

Mike Folk (The HDF Group)

Robert McGrath (NCSA)

Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)

Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA)

Jennifer Adams (GrADS)

Dave Brown (UCAR)

HDF and HDF-EOS Workshop XII


Hdf5 opendap project update and demo

Acknowledgement

  • This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 

HDF and HDF-EOS Workshop XII


  • Login