slide1
Download
Skip this Video
Download Presentation
OPeNDAP (The Open-source Project for a Network Data Access Protocol) BOM Tutorial Introduction October 15, 2007 [email protected]

Loading in 2 Seconds...

play fullscreen
1 / 55

OPeNDAP - PowerPoint PPT Presentation


  • 243 Views
  • Uploaded on

OPeNDAP (The Open-source Project for a Network Data Access Protocol) BOM Tutorial Introduction October 15, 2007 [email protected] Distributed Oceanographic Data System (DODS). Conceived in 1993 at a workshop held at URI. Objectives were:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'OPeNDAP' - Michelle


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

OPeNDAP(The Open-source Project for a Network Data Access Protocol) BOM Tutorial IntroductionOctober 15, [email protected]

BOM, Melbourne, VIC 20071015 (Fox)

distributed oceanographic data system dods
Distributed Oceanographic Data System (DODS)
  • Conceived in 1993 at a workshop held at URI.
  • Objectives were:
    • to facilitate access to PI held data as well as data held in national archives and
    • to allow the data user to analyze data using the application package with which he or she is the most familiar.
  • Basic system designed and implemented in 1993-1995 by Gallagher and Flierl with NASA and NOAA funding.
  • From 1994 to present it has been extended with NASA, NOPP, NSF and NOAA funding.

BOM, Melbourne, VIC 20071015 (Fox)

slide3

Some Definitions

DAP = Data Access Protocol

  • Model used to describe the data;
  • Request syntax and semantics; and
  • Response syntax and semantics.

OPeNDAP

  • The software;
  • Numerous reference implementations;
  • Core/libraries and services (servers and clients).

OPeNDAP Inc.

  • OPeNDAP is a 501.c(3) non-profit corporation;
  • Formed to maintain, evolve and promote the discipline neutral DAP that was the DODS core infrastructure.

BOM, Melbourne, VIC 20071015 (Fox)

slide4

Some Definitions

Syntax

  • The computer representation of a data object - the data types and structures at the computer level; e.g.,
  • T is a floating point array of 20 by 40 elements.

Semantics

  • The information about the contents of an object; e.g.,
  • T is sea surface temperature in degrees Celsius for a certain region of the Earth.

BOM, Melbourne, VIC 20071015 (Fox)

considerations with regard to the development of dap and opendap

Considerations with regard to the development of DAP and OPeNDAP

  • Many data formats
  • Many different client types

Many data providers

  • Many different semantic representations of the data
  • Many different security requirements

BOM, Melbourne, VIC 20071015 (Fox)

slide6

Broad Vision

A world in which a single data access protocol is used for the exchange of data between network based applications regardless of discipline.

A layer above TCP/IP providing for syntactic and semantic consistency not available in existing protocols such as FTP.

BOM, Melbourne, VIC 20071015 (Fox)

slide7

Practical Considerations

The broad vision:

  • Is syntactically achievable, but
  • Is not semantically achievable, at least not in the near term.

BOM, Melbourne, VIC 20071015 (Fox)

slide8

OPeNDAP Inc. Mission Statement

To maintain, evolve and promote a data access protocol (DAP) and reference implementation software (OPeNDAP) for the syntactically consistent exchange of data over the network.

The DAP should provide syntactic interoperability across disciplines and allow for semantic interoperability within disciplines.

BOM, Melbourne, VIC 20071015 (Fox)

slide9

OPeNDAP Inc. Vision Statement

To achieve the mission:

  • OPeNDAP Inc. will be non-profit.
    • Easier to obtain federal funds.
    • The DAP is more likely to be adopted.
  • OPeNDAP software will be open source.
    • More likely to be adopted.
    • Need community contributions to software.
  • OPeNDAP Inc. will mix implementation with research.
    • Implementation - to encourage use.
    • Research - to keep the protocol current.

BOM, Melbourne, VIC 20071015 (Fox)

slide10

OPeNDAP Inc. Vision Statement(cont)

  • OPeNDAP Inc. will rely primarily on federal funding.
    • Unlikely to obtain private funding for middleware.
    • Development to be use case driven.
    • Aligned with Vision/Mission.
  • Strategic direction will be sought from an Advisory Board consisting of data system experts and with input from you the community of developers.
  • OPeNDAP Inc. will seek partners.
  • OPeNDAP Inc. will utilize community working groups to develop ‘standards’ related to the DAP, OPeNDAP

BOM, Melbourne, VIC 20071015 (Fox)

slide11

The Data Access Protocol (DAP)

  • The DAP has been designed to be as general as possible without being constrained to a particular discipline or world view.
  • The DAP is a discipline neutral data access protocol; it is being used in astronomy, medicine, earth science,…
  • Provides data format and location, and data organization transparency
  • Is metadata neutral

BOM, Melbourne, VIC 20071015 (Fox)

dap comparisons
DAP comparisons
  • File-based
    • GridFTP/FTP
    • HTTP
    • SRB
  • Service-based
    • Open-Geospatial Consortium, WCS, WMS, WFS, …
    • Virtual Observatory (Astronomy), SIAP, SSAP, STAP,…

BOM, Melbourne, VIC 20071015 (Fox)

the astronomy approach data types as a service

Limited interoperability

Lightweight semantics

Limited meaning, hard coded

Limited extensibility

The Astronomy approach; data-types as a service
  • VOTable
  • Simple Image Access Protocol
  • Simple Spectrum Access Protocol
  • Simple Time Access Protocol

VO App2

VO App3

VO App1

OGC: {WFS, WCS, WMS} and SWE {SOS, SPS, SAS}

use the same approach

VO layer

DBn

DB2

DB3

… … … …

DB1

BOM, Melbourne, VIC 20071015 (Fox)

who is using dap opendap
Who is using DAP/ OPeNDAP?
  • Science examples
    • PMEL with their Tsunami inundation modeling
    • Ocean regional modelers to extract open boundary conditions
    • Visualization of data sets using MATLAB/IDL/…
  • Service examples
    • Live Access Server
    • Mapserver – OGC services and OPeNDAP data access (future)
    • Digital Library Service - metadata and catalogue info

BOM, Melbourne, VIC 20071015 (Fox)

data access protocol dap2 current
Data Access Protocol (DAP2) - Current
  • DAP2 currently a NASA/ESE ‘Standard’
  • Current servers implement DAP2

DAP3

  • DAP 2 + XML responses (implemented)

BOM, Melbourne, VIC 20071015 (Fox)

dap4 late 2007
DAP4 - Late 2007 (?)
  • DAP4 improvements over DAP3:
    • Additional datatypes
      • Swath
      • Blob - GIF, MPEG,…
    • Additional functionality
      • Check sum
      • Modulo
    • The additional datatypes will enable the DAP to be used in a wider variety of circumstances and are a direct response to users’ requests.

BOM, Melbourne, VIC 20071015 (Fox)

what dap means to me
What DAP means to me
  • Data access and transport
  • Response types: DAP objects versus file type
    • A DAP URL is essentially an HTTP URL with additional restrictions placed on the abs-path component.
    • DAP2-URL = "http://" host [ ":" port ] [ abs-path]
      • abs-path = server-path data-source-id [ "." ext[ "?" query ] ]
      • server-path = [ "/" token ]
      • data-source-id = [ "/" token ]
      • ext = "das" | "dds" | "dods"
    • The server-path is the pathname to the server, whereas data-source-id is the pathname to the data.

BOM, Melbourne, VIC 20071015 (Fox)

opendap v3 architecture
OPeNDAP V3 Architecture

Client

Cgi style access

Data

  • CGI-style access
    • Uses web server
    • HTTP protocol
    • Several request and response types
    • Reads data files, Databases, et c., returns info
    • May return DAP2 objects or other data
    • Client can be application, web browser or specialized server/service

BOM, Melbourne, VIC 20071015 (Fox)

opendap v4 hyrax architecture
OPeNDAP V4 (Hyrax) Architecture

Client

OLFS

BES

Data

  • OPeNDAP Lightweight Front end Server (OLFS)
    • Receives requests and asks the BES to fill them
    • Uses Java Servlets
    • Does not directly ‘touch’ data
    • Multi-protocol
  • Back End Server (BES)
    • Reads data files, Databases, et c., returns info
    • May return DAP2 objects or other data
    • Does not require web server

BOM, Melbourne, VIC 20071015 (Fox)

binaries generated
Binaries Generated

There are approximately 80 binaries built on a nightly basis. They are built for the following platforms/operating systems:

  • Linux
    • FC4
    • FC5
  • MacOS-X (universal binaries when possible)
  • Windows XP, win32
  • Java 1.5 (Tomcat 5.5)
  • IRIX (in four variants), Solaris, AIX, OSF

BOM, Melbourne, VIC 20071015 (Fox)

communication
Communication
  • Website (http://www.opendap.org and test.opendap.org)
  • SVN - Code repository (http://scm.opendap.org:8090/trac/browser/)
  • Trac - Task/milestone repository (http://scm.opendap.org:8090/trac/)
  • Telecons
    • Management - ~ Weekly
    • Developers - Weekly on Monday 11am MT
  • Twiki -> MediaWiki, for
    • Management
    • Developers
    • Coming soon http://docs.opendap.org (documentation on-line in MediaWiki form)
  • [email protected] e-mail list
  • [email protected] for announcements (low vol.)

BOM, Melbourne, VIC 20071015 (Fox)

releases support
Releases/Support

BOM, Melbourne, VIC 20071015 (Fox)

opendap and security
OPeNDAP and security
  • Both administrators and developers must consider security aspects of OPeNDAP software
  • First developer telecon each month addresses security
  • This will be addressed during this week

BOM, Melbourne, VIC 20071015 (Fox)

opendap and authentication
OPeNDAP and authentication
  • ESG II/III
  • Australian Access Federation (AAF)
  • .htpasswd
  • Tomcat authentication

BOM, Melbourne, VIC 20071015 (Fox)

esg security services
ESG Security Services

BOM, Melbourne, VIC 20071015 (Fox)

esg security services client interaction
ESG Security Services - Client Interaction

BOM, Melbourne, VIC 20071015 (Fox)

opendap community working groups
OPeNDAP Community Working Groups
  • Modeled after best practices of IETF, W3, OGC, IEEE, ISO, and others
  • Working Groups:
    • Authentication
    • Security
    • Server-side Functions
    • Virtualization (Aggregation)
    • Server-side processing
    • Geospatial Interoperability
    • Hyrax and *DS (TDS, GDS, FDS, etc.)
    • Semantics
    • DAPPER
    • netCDF C++ client
    • Response types
    • Metrics
    • Asynchronous transactions
    • DAP4
    • Relational Database access via DAP
  • Wiki (http://docs.opendap.org/index.php/Working_Groups)

BOM, Melbourne, VIC 20071015 (Fox)

opendap community working groups28
OPeNDAP Community Working Groups
  • Terms of Reference http://docs.opendap.org/index.php/Terms_of_Reference
  • Abridged version:
    • Each OPeNDAP Working Group is established to apply members\' expertise in their focus area to produce specific deliverables or outcomes
    • Types: Software and Documentation
    • Minimum 3 people
    • 6 month time-frame

BOM, Melbourne, VIC 20071015 (Fox)

slide29

OPeNDAP System Elements

The OPeNDAP data access protocol is used by a variety of system elements.

  • Clients
  • Browser Interfaces
  • Data System Integrators (ODC)
  • Servers
  • Processing Servers
  • Aggregating Servers - OPeNDAP chains
  • Ancillary Information Services

BOM, Melbourne, VIC 20071015 (Fox)

slide30

Clients

  • Clients make requests and receive responses via the DAP.
  • Clients convert data from the OPeNDAP data model to the form required in the client application.

BOM, Melbourne, VIC 20071015 (Fox)

slide31

IDL

Client

pyDAP

pyDAP

netCDF Java

netCDF C

ArcGIS

ArcGIS

Ferret

GrADS

IDV

VisAD

ncBrowse

IDL

Access

Excel

OPeNDAP Clients

Internet

NCL

Client

Matlab

Client

Web

Browser

OPeNDAP

Data

Connector

NCL

Matlab

BOM, Melbourne, VIC 20071015 (Fox)

ocapi
OCAPI
  • A pure OPeNDAP C API (OCAPI) for the client-side
  • Applications:
    • DAP-aware ‘commands’ for commercial analysis programs (e.g., IDL)
    • Scripting tools (e.g., Perl)

BOM, Melbourne, VIC 20071015 (Fox)

slide33

OPeNDAP System Elements

The OPeNDAP data access protocol is used by a variety of system elements.

  • Clients
  • Browser Interfaces
  • Data System Integrators (ODC)
  • Servers
  • Processing Servers
  • Aggregating Servers - OPeNDAP chains
  • Ancillary Information Services

BOM, Melbourne, VIC 20071015 (Fox)

browser interfaces
Browser interfaces

BOM, Melbourne, VIC 20071015 (Fox)

slide35

OPenDAP Data

Connector

- a Data System Integrator

GCMD

GFDL

netCDF

URI

HDF

GSFC

Binary

Matlab

NVODS

OPeNDAP

Access

IDL

ncBrowse

IDV

Ferret

VisAD

GrADS

Excel

ODC

BOM, Melbourne, VIC 20071015 (Fox)

slide36

OPeNDAP System Elements

The OPeNDAP data access protocol is used by a variety of system elements.

  • Clients
  • Browser Interfaces
  • Data System Integrators (ODC)
  • Servers
  • Processing Servers
  • Aggregating Servers - OPeNDAP chains
  • Ancillary Information Services

BOM, Melbourne, VIC 20071015 (Fox)

slide37

Servers

  • Servers receive requests and provide responses via the DAP.
  • Servers convert the data from the form in which they are stored to the DAP.
  • Servers provide for subsetting of the data and more.

BOM, Melbourne, VIC 20071015 (Fox)

slide38

netCDF

DSP

Tables

Data

Data

Data

Data

General

netCDF

JGOFS

DSP

OPeNDAP Servers

CDM

Flat

Binary

ESML

HDF4

HDF5

SQL

FITS

CDF

CEDAR

Data

Data

Data

Data

Data

Data

Data

HDF5

FITS

FreeForm

HDF4

JDBC

CDF

CEDAR

Internet

BOM, Melbourne, VIC 20071015 (Fox)

slide39

ESG

FDS

GDS

DAPPER

CODAR

pyDAP

pyDAP

Data

Data

Data

Data

Data

Data

Data

netCDF

OPeNDAP

netCDF

OPeNDAP

GRIB

BUFR

OPeNDAP

netCDF

OPeNDAP

CODAR

General

General

TDS

TDS

Data

Data

netCDF

OPeNDAP

netCDF

OPeNDAP

OPeNDAP Servers

(specialized processing)

Internet

BOM, Melbourne, VIC 20071015 (Fox)

slide40

Servers

  • Servers may also provide other services
    • Directory traversal.
  • Browser-based form to build URL.
  • Ascii or other representations of data.
  • Metadata associated with the data.
  • Server side functions.

BOM, Melbourne, VIC 20071015 (Fox)

slide41

TDS

JGOFS

CODAR

Data

Data

Data

ESG

FDS

GDS

DAPPER

pyDAP

netCDF

OPeNDAP

General

CODAR

Data

Data

Data

Data

Data

netCDF

OPeNDAP

netCDF

OPeNDAP

GRIB

BUFR

OPeNDAP

netCDF

OPeNDAP

General

OPeNDAP Aggregation Servers

Internet

BOM, Melbourne, VIC 20071015 (Fox)

slide42

The Aggregation Server: An Example

Local

OPeNDAP

HTML, GIF

netCDF Data Set

DSP Data Set

File

File

File

File

File

File

DSP

Aggregation

Server

Matlab

Client

Matlab

BOM, Melbourne, VIC 20071015 (Fox)

opendap s hyrax server4
OPeNDAP’s Hyrax (‘Server4’)
  • Uses a modular architecture to support different application-level protocols
    • Data access using DAP2 (DAP3)
    • Catalogs using THREDDS
    • Browsing using HTML and ASCII
  • Modules for data access
    • Different file types
    • Potential for database and scripting
  • Modules for commands
    • Commands provide varying operations for different protocols

BOM, Melbourne, VIC 20071015 (Fox)

opendap v4 hyrax architecture44
OPeNDAP V4 (Hyrax) Architecture

Client

OLFS

BES

Data

  • OPeNDAP Lightweight Front end Server (OLFS)
    • Receives requests and asks the BES to fill them
    • Uses Java Servlets
    • Does not directly ‘touch’ data
    • Multi-protocol
  • Back End Server (BES)
    • Reads data files, Databases, et c., returns info
    • May return DAP2 objects or other data
    • Does not require web server

BOM, Melbourne, VIC 20071015 (Fox)

slide45

Info output

HTML form

ASCII output

OPeNDAP Lightweight Front end Server

GridFTP

DAP2

Request Formulation**

Request from client

HTTP

DAP2

DAP2 (GridFTP, HTTP)

BES

SOAP-DAP (HTTP)

THREDDS

Response to client

BOM, Melbourne, VIC 20071015 (Fox)

slide46
BES

BES Framework

Network Protocol and

Process start/stop

activities

PPT*

Initialization/

Termination

DAP2

Access

Data

Catalogs

BES Commands/

XML Documents

Commands**

NetCDF3

HDF4

FreeForm

Data Store Interfaces

*PPT is built in (other protocols)

**Some commands are built in

Data

Data

Data

BOM, Melbourne, VIC 20071015 (Fox)

additional material
Additional material

BOM, Melbourne, VIC 20071015 (Fox)

opendap relies on projects
OPeNDAP relies on projects
  • To guide user-based requirements for application needs as well as OPeNDAP core development (use cases)
  • To provide maintenance and evolution of the core software and documentation
  • Currently: 7 active projects, covers next ~2-3 years, 2 pending proposals
  • ……

BOM, Melbourne, VIC 20071015 (Fox)

success
Success

What constitutes success of the OPeNDAP mission?

Adoption of the DAP across a broad range of disciplines with extensive use in several of these.

In order to achieve this the DAP must do the following:

  • It must be sufficiently flexible, all encompassing, etc. that it can be used across a broad range of data types.
  • Its implementation must be robust, secure, easy to use, provide for a broad range of services, etc.
  • The funding stream must be robust.
  • Active and engaged developer and user community

BOM, Melbourne, VIC 20071015 (Fox)

risks
Risks

It is still the case that

  • Some other data access protocols are seen as more attractive regardless of whether or not they are, or
  • Other data access protocols are developed because their community is not aware of OPeNDAP or of what its capabilities are.
  • We will compare and contrast some of these today

BOM, Melbourne, VIC 20071015 (Fox)

to succeed
To Succeed

OPeNDAP must make sure that:

  • It’s (server and client) capabilities are well known across a broad range of disciplines.
  • The data model is inclusive.
  • The implementation is robust and meets user’s needs.
  • The DAP coexists with other protocols.
  • It has a robust funding base.
  • It has extensive documentation.

BOM, Melbourne, VIC 20071015 (Fox)

robust funding
Robust Funding?
  • What is an appropriate level of activity (funding) for OPeNDAP? What should OPeNDAP be doing?
    • Core only? + Clients and servers? + Demonstration projects?

Should OPeNDAP be a small staff (core only) or…?

What priority should be assigned to the elements currently being developed? Which of these go beyond the core?

    • DAP4
    • AIS - metadata consistency
    • THREDDS - Catalogs
  • Toolkits - Matlab, IDL,…
  • Grid capabilities
  • OTHERS?

BOM, Melbourne, VIC 20071015 (Fox)

ancillary information service
Ancillary Information Service
  • Current capability: Attributes only
  • Client-side only
  • Local and remote resources
  • Local resource databases

The AIS enables users to augment the metadata for a data source in a controlled way without requiring write access to the original data. By using the DAP, users are also isolated from data format issues.

BOM, Melbourne, VIC 20071015 (Fox)

ais enhancements
AIS enhancements
  • Remote resource databases
  • AIS server
  • AIS for variables

These enhancements will greatly expand the usefulness of the AIS: Remote resource databases and an AIS server will enable third-party ‘AIS sites’ which may be sponsored by project offices or institutions, et cetera.

AIS for variables will enable adding metadata which are stored as ‘data.’

BOM, Melbourne, VIC 20071015 (Fox)

proposed ais server
Proposed AIS Server

0

1

Client linked

w/DAP

Software

AIS

Server

Data

Source

3

2

AIS

Resource

0. Client requests metadata from the AIS server (which appears no different from any other DAP server).

1. The AIS server gets metadata from data source

2. The AIS server gets matching the AIS resource using the AIS database and merges it into the metadata.

3. The AIS server returns resulting the metadata object.

BOM, Melbourne, VIC 20071015 (Fox)

ad