Opendap 4 data server hyrax
This presentation is the property of its rightful owner.
Sponsored Links
1 / 24

OPeNDAP 4 Data Server – Hyrax PowerPoint PPT Presentation


  • 77 Views
  • Uploaded on
  • Presentation posted in: General

OPeNDAP 4 Data Server – Hyrax. James Gallagher and Nathan Potter 21 Feb 2007. But first…. These are Hyrax. “Despite the unexceptional appearance of modern hyraxes, the order has a remarkable prehistoric lineage and hyraxes may be the closest living relatives of the elephant.” WikiPedia.

Download Presentation

OPeNDAP 4 Data Server – Hyrax

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Opendap 4 data server hyrax

OPeNDAP 4 Data Server – Hyrax

James Gallagher and Nathan Potter

21 Feb 2007

OPeNDAP Developer’s Workshop Feb 21-23 2007


But first

But first…

  • These are Hyrax

“Despite the unexceptional appearance of modern hyraxes, the order has a remarkable prehistoric lineage and hyraxes may be the closest living relatives of the elephant.” WikiPedia.

OPeNDAP Developer’s Workshop Feb 21-23 2007


Overview

Overview

  • Goals of the Hyrax Project

  • Introduction to customization

  • Discussion topic

OPeNDAP Developer’s Workshop Feb 21-23 2007


Goals of the hyrax project

Goals of the Hyrax Project

  • Integrate THREDDS catalogs with DAP

  • Improve security of our data server

  • Reduce work to maintain the server

  • Reuse the existing server software

  • Integrate work with/for Earth System Grid (ESG) back into the OPeNDAP software

  • Improve the server’s performance

OPeNDAP Developer’s Workshop Feb 21-23 2007


Integrate thredds

Integrate THREDDS

  • Requirements:

    • Standard representation for catalogs

    • Separation of the logical from the physical data organization

    • A no-hands mode » Fully automatic catalogs

    • Graceful transition from custom to automatic catalogs

    • Full THREDDS features – not stripped down

    • Must use Unidata’s THREDDS software

  • Hyrax 1.0.2 has met these.

OPeNDAP Developer’s Workshop Feb 21-23 2007


Improve the server s security

Improve the Server’s Security

  • The original data server design was hard to make ‘hack proof’

    • Common Gateway Interface (CGI)

    • Data handlers were run as a subprocess in a protected shell

    • Main CGI modules written in Perl

  • As the server grew in complexity, these became more pressing issues.

  • Improvements

    • Perl/CGI » Java/Servlet

    • Subprocess/shell » Specialized IPC

OPeNDAP Developer’s Workshop Feb 21-23 2007


Reduce maintenance and reuse existing software

Reduce Maintenance and Reuse Existing Software

  • The Perl software was becoming unwieldy – Reimplementation was needed whether in Perl, Java, …

  • Considerable effort has gone into writing our data handlers for netCDF, …, and we could not rewrite them.

  • Additional code for the handlers had to be minimal (for the most complex handler - asciival, it’s size increased by 21%, format handlers were just half that value)

OPeNDAP Developer’s Workshop Feb 21-23 2007


Integrate esg work

Integrate ESG Work

  • ESG II had developed a flexible server for the DAP

    • Its capabilities far out stripped the CGI server

    • Several extensions to OPeNDAP software were needed

  • The changes/extensions to the OPeNDAP software were merged back into our copies of the code and both projects resumed working from the same version of the software.

OPeNDAP Developer’s Workshop Feb 21-23 2007


Improve performance

Improve Performance

  • The new server architecture eliminates basic design problems.

    • Fewer subprocess created, issues with Perl, …

  • Response handlers in the BES pass binary objects, not the objects’ network representation

  • Better use of HTTP (e.g., persistent connections now supported)

OPeNDAP Developer’s Workshop Feb 21-23 2007


Recap goals of hyrax

Recap: Goals of Hyrax

  • A higher performance server

  • Provides true THREDDS catalogs

  • A high level of data-handling code reused

  • A more secure design

  • A more maintainable design that can evolve gracefully

OPeNDAP Developer’s Workshop Feb 21-23 2007


Introduction to customization

Introduction to Customization

  • The OPeNDAP Data Server was designed to be extended.

  • Hyrax - can be Extended by:

    • Writing a handlers for the front end

    • Writing a handlers for the back end

    • Directly access the back end

OPeNDAP Developer’s Workshop Feb 21-23 2007


Hyrax architecture

OLFS

Java

Servlet Engine

Hyrax Architecture

BES Commands

DAP2

BES

THREDDS

HTML

XML- encapsulated object

Unix Daemon

Optional THREDDS catalogs

File system with data files,

SQL Database, …

OPeNDAP Developer’s Workshop Feb 21-23 2007


Adding a new bes handler

Adding a new BES Handler

  • Add support for a new data format or data source (relational data base)

  • Build a gateway to another data system (such as a WCS server)

  • Support new response types (such as netCDF 3 files)

  • Add new commands

  • Patrick West will talk about this next.

OPeNDAP Developer’s Workshop Feb 21-23 2007


Direct access to the bes

Direct access to the BES

  • Use this to build a connection-oriented system

  • Build new front end systems

  • And more…

  • Jose Garcia will talk about this following Patrick.

OPeNDAP Developer’s Workshop Feb 21-23 2007


Adding a new olfs handler

Adding a new OLFS Handler

  • OLFS

    • Dispatch activity abstracted

    • Dispatch figures out what they want

    • Uses a handler to figure out how to do it

OPeNDAP Developer’s Workshop Feb 21-23 2007


Inside olfs

Inside OLFS

Dispatch

DispatchHandler

OPeNDAP Developer’s Workshop Feb 21-23 2007


Inside olfs1

Inside OLFS

Dispatch

DispatchHandler

GET

POST

Network Interface

OPeNDAP Developer’s Workshop Feb 21-23 2007


Inside olfs2

Inside OLFS

Dispatch

DispatchHandler

GET

POST

Network Interface

Uniform Dispatch Interface

OPeNDAP Developer’s Workshop Feb 21-23 2007


Inside olfs3

Inside OLFS

Dispatch

DispatchHandler

GET

HttpDispatchHandler

ThreddsDispatchHandler

POST

SoapDispatchHandler

Network Interface

Uniform Dispatch Interface

OPeNDAP Developer’s Workshop Feb 21-23 2007


Inside olfs4

Dispatch

DispatchHandler

GET

HttpDispatchHandler

DAS

DDS

catalog

ThreddsDispatchHandler

POST

SoapDispatchHandler

SOAP

Inside OLFS

Network Interface

Uniform Dispatch Interface

OPeNDAP Developer’s Workshop Feb 21-23 2007


Inside olfs5

Dispatch

DispatchHandler

GET

HttpDispatchHandler

BES

DAS

DDS

catalog

ThreddsDispatchHandler

Crawlable…

POST

SoapDispatchHandler

BES

SOAP

Network Interface

Uniform Dispatch Interface

Various sources of information

Inside OLFS

OPeNDAP Developer’s Workshop Feb 21-23 2007


Summary

Summary

  • BES Handler: Add support for a new data format, response type, commands or to build a gateway

  • OLFS Handler: Build support for a new network protocol while (re)using the existing software and/or interact with a new data source

  • Use the BES directly: Build new front–ends (web portals, LAN-based systems)

OPeNDAP Developer’s Workshop Feb 21-23 2007


Recap hyrax architecture

OLFS

Java

Servlet Engine

Recap: Hyrax Architecture

BES Commands

DAP2

BES

THREDDS

HTML

XML- encapsulated object

Unix Daemon

Optional THREDDS catalogs

File system with data files,

SQL Database, …

OPeNDAP Developer’s Workshop Feb 21-23 2007


Discussion topic how could hyrax support the wcs protocol

Discussion Topic: How Could Hyrax Support the WCS Protocol?

  • A BES Handler could be written to support new response formats (i.e., the well-known binary formats)

  • A BES command set could be written for operations unique to WCS

  • An OLFS DispatchHandler could be built to use the new BES commands

OPeNDAP Developer’s Workshop Feb 21-23 2007


  • Login