muqun yang and hyo kyung lee thg james gallagher opendap inc n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
HDF5 OPeNDAP Project PowerPoint Presentation
Download Presentation
HDF5 OPeNDAP Project

Loading in 2 Seconds...

play fullscreen
1 / 42

HDF5 OPeNDAP Project - PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on

MuQun Yang and Hyo -Kyung Lee (THG) James Gallagher ( OPeNDAP , Inc.). HDF5 OPeNDAP Project. Question of the Day. How can I get One Sub from SUBWAY? C hicken teriyaki with no onion . How can I get One Subset of EOS data image from NASA?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'HDF5 OPeNDAP Project' - barto


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide2

Question of the Day

  • How can I get One Sub from SUBWAY?
    • Chicken teriyaki with no onion
  • How can I get One Subset of EOS data image from NASA?
    • Today’s satellite image right above the roof
slide3

Subway

How can I get One Sub from SUBWAY?

  • Easy: anywhere / accepts Visa|MC
  • Efficient: your own recipe
  • Cool: eat fresh and look good
slide4

Dumb-way

How can I get One Subset of EOS data image from NASA?

  • Difficult: bring a USB drive and visit NASA
  • Inefficient: download 10 Gbyte and search for 10 byte dataset
  • Not cool: wrong or useless dataset
slide5

DAP-way!

How can I get One Subset of EOS data image from NASA?

  • Easy: anywhere / accepts IE|FF
  • Efficient: WYSIWYG – less fat!
  • Cool: Visualization clients – see fresh!
slide6

DAP-way Implementation

  • OPeNDAP: Free DAP-way Software
  • Server-Client Model APIs
  • Widely used by Earth Science Community
slide7

OPeNDAP server sites

OPeNDAP

  • Market Dominance:
  • http://www.opendap.org

Courtesy of OPeNDAP, Inc.

slide8

Flat

Binary

CODAR

General

netCDF

HDF4

Matlab

DSP

Tables

SQL

FITS

CDF

CEDAR

Data

Data

Data

Data

Data

Data

Data

Data

Data

Data

Data

Data

ESML

netCDF

Matlab

JGOFS

FITS

FreeFrom

CODAR

HDF4

DSP

JDBC

CDF

CEDAR

OPeNDAP Servers

Courtesy of OPeNDAP, Inc.

slide9

IDL

Client

Matlab

Client

netCDF Java

netCDF C

Ferret

GrADS

IDV

VisAD

ncBrowse

Matlab

IDL

Access

Excel

OPeNDAP Clients

Courtesy of OPeNDAP, Inc.

slide10

How OPeNDAP works

NetCDF

OPeNDAP

NetCDF Server

HDF4

OPeNDAP

Client

OPeNDAP

HDF4 Server

slide11

Architecture of OPeNDAP Server

  • Server 3 (CGI)
    • C/C++ based HTTP Server
    • Insecure / Inflexible
  • Server 4 (Hyrax)
    • Java based TomCatServlet Engine
    • Secure / Flexible
server 4 hyrax architecture
Server 4 (Hyrax) Architecture

OLFS

BES

Java

Servlet Engine

Unix Daemon

Server 3 (CGI) Architecture

HTTP Server

BES Commands

DAP

XML, GIS, KML

HTML

XML- encapsulated object

Data

Store

Optional catalogs:

XML, GIS, KML

File system with data files like HDF, NetCDF and SQL Database, …

http://docs.opendap.org/index.php/Hyrax

Courtesy of OPeNDAP, Inc.

slide13

Example Usage

HDF4

OPeNDAP

Client

OPeNDAP

HDF4 Server

Actual Content of Data

Semantic Meaning of Data

Syntactic Structure of Data

OPeNDAP Visualization Client

slide14

What about HDF5?

NetCDF

OPeNDAP

NetCDF Server

HDF4

OPeNDAP

Client

OPeNDAP

HDF4 Server

HDF5

OPeNDAP

HDF5 Server

slide15

Why Important?

  • HDF5: NASA EOS / NPOESS
  • NASA’s MISSION: ACCESS!
  • Our MISSION: Build DAP-way Bridge
slide16

Goals

  • 1st: Transform HDF5 access in DAP-way
  • 2nd: Yet preserve the beauty of HDF5
slide17

OPeNDAP HDF5 Server

Goal: Transform HDF5 access in DAP-way

  • Map HDF5 Datatypes in DAP-way
  • Plus, tweaks for OPeNDAP Clients
  • Plus, tweaks for HDF-EOS Files
slide18

Mapping HDF5 in DAP-way

  • Prototype server in 2001
  • NASA Grant from Nov. 2006
  • Support for Compound Datatype
  • Support for Group
  • Support for References / Links
  • First Product Release in Mar. 2008
slide19

OPeNDAP HDF5 Server

Goal: Transform HDF5 access in DAP-way

  • Map HDF5 Datatypes in DAP-way
  • Plus, tweaks for OPeNDAP Clients
  • Plus, tweaks for HDF-EOS Files
slide20

Subway Customers

  • I want 3 inch-long sub
  • I need Wasabi sauce
  • I eat only wheat bread
  • I eat only meat balls
slide21

OPeNDAP Clients

Not all OPeNDAP clients are created equal!

  • I hate foot-long variable names.
  • I need special attributes on dataset.
  • I care pre-defined Grid data type.
  • I care only well-formed attributes.
slide22

Tweaks for OPeNDAP Clients

  • Two configuration options
  • --enable-short-name = cut bread
  • --enable-CF = put Wasabi sauce
slide23

OPeNDAP HDF5 Server

Goal: Transform HDF5 access in DAP-way

  • Map HDF5 Datatypes in DAP-way
  • Plus, tweaks for OPeNDAP Clients
  • Plus, tweaks for HDF-EOS5 Files
slide24

Subway Customers

  • I want 3 inch-long
  • I need Wasabi sauce
  • I eat only wheat bread
  • I eat only meat balls
slide25

Subway Customers - OK

  • I eat only wheat bread
  • I eat only meat balls

Subway Suppliers Problem

  • No breads, only wheat!
  • No meat balls, only meat!
slide26

HDF EOS

Not all HDF5 files are created for DAP-way!

  • Grid with No Dimension data
    • Clients expect Grid with Dimension data
  • Metadata Attribute in Two Strings
    • Clients expect One Structured Format
slide27

Tweaks for HDF-EOS

  • Two more configuration options
  • --enable-eos-grid = bake bread
  • --enable-eos-meta = make balls
slide28

OPeNDAP HDF5 Server

Goal: Transform HDF5 access in DAP-way

  • Map HDF5 Datatypes in DAP-way
  • Plus, tweaks for OPeNDAP Clients
  • Plus, tweaks for HDF-EOS5 Files
slide29

Day After Server Tweaks

  • Finally, Happy Clients!

ncBrowse

Ferret

MATLAB

NCL

IDV

GrADS

ODC

slide30

Live Demo

Ozone concentration level over the south pole

slide31

Goals

  • 1st: Transform HDF5 access in DAP-way
  • 2nd: Yet Preserve the beauty of HDF5
slide32

HDF5 Served in DAP-way

(Yet Preserve the beauty of HDF5)

  • There are some things money can’t buy. For everything else, there is MasterCardTM.

Minimize

  • There are some things DAP can’t serve. For everything else, there is _________?

Maximize

slide33

Some Things

(that OPeNDAP HDF5 server can’t serve)

  • Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types
  • Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5
slide34

For Everything Else

  • Let there be HDF5-friendly OPeNDAP client library!
  • Package and Deliver in DAP-way first.
  • Then, let the client library handle it.
  • Concept of Trojan Horse
slide35

Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.

Example: Group in HDF5

HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

slide36

Example: Reference in HDF5

  • Object / Regional Reference
  • Map to DAP URL at server
  • No de-referencing of URL at client library
  • Important for NPOESS
slide37

HDF5-Friendly OPeNDAP Client Library

No Latitude and Longitude

  • One more reason: Tame Clients
  • Only ODC can display Swath properly.
  • Easy but Evil OPeNDAP NC Client Library!
    • easy: nc style API
    • evil: fixed dimensional attributes
  • Our client library must be Easy but Good

Courtesy of NASA

slide38

NC-Friendly OPeNDAP Client Library

HDF5-Friendly OPeNDAP Client Library

???

View NetCDF

View G/R/Swath

Group/Ref/Swath

NetCDF

View HDF5

HDF5

GrADS

GrADS

OPeNDAP

HDF5 Server

OPeNDAP

NC Server

liboc-dap

libnc-dap

dapserver

libdap

slide39

Summary

  • DAP-way access of HDF5
    • Is Easy / Efficient / Cool
    • Loses some things
    • Requires HDF5-friendly OPeNDAP Client Library
  • HDF5-friendly OPeNDAP Client Library
    • Serves HDF5 better
    • Tames evil visualization clients
slide40

Future Work

  • HDF5 DAP Mapping Document
  • Finish HDF5-friendly OPeNDAP Client Library prototype
  • Test it on GrADS and display Swath
slide41

Project Website

  • http://hdfdap.hdfgroup.uiuc.edu/joomla
  • Beta preview
  • Feedbacks are more than welcome!
slide42

Credits

Mike Folk (THG)

Robert McGrath (NCSA)

Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)

Christopher Lynnes, James Johnson, Denis Nadeau (NASA)

Jennifer Adams (GrADS)

Dave Brown (UCAR)