190 likes | 195 Views
OPeNDAP/Hyrax Interfaces. James Gallagher. What are...?. OPeNDAP: A not-for-profit corporation which steers the growth of the DAP. All software it develops is open source.
E N D
OPeNDAP/Hyrax Interfaces • James Gallagher
What are...? • OPeNDAP: A not-for-profit corporation which steers the growth of the DAP. All software it develops is open source. • DAP: Data Access Protocol - a simple request-response protocol which uses HTTP to move structured (scientific) data from a server to a client. • Hyrax: A data server developed by OPeNDAP that supports DAP.
Types of Interfaces Application Server Software libraries Component
Application Level • DAP Requests made with a URL • DAP Responses returned as MIME documents • DAP is stateless; implementation using REST • DAP response form is a ‘universal representation’ (canonical) for data - it’s generated by a server (nominally) on the fly without any need to modify the original data. • DAP servers for over a dozen very different formats exist • DAP Servers also support ‘Web’ features • Other protocols that work with the Web (e.g., THREDDS) easily coexist with DAP • See Data Access Protocol (DAP), version 2
Server: Hyrax • Front end (OLFS); Java; Processes requests and uses the BES • Back end (BES); C++; Builds responses and responds to OLFS commands • N-Tier architecture • OLFS and BES are extensible • See Hyrax
OLFS OLFS BES BES OLFS OLFS BES BES WCS BES Hyrax: Many Geometries Simplest... one host or two One OLFS talks to many BES daemons Many OLFS front-ends talk to one BES
Components: BES • Format handlers: Read data • Response handlers: Build new responses • Reporters: Record usage information • Aggregators: Build aggregate responses • Implement C++ classes • New modules are loaded at runtime • See Hyrax: For Developers, Extending the BES
Components: OLFS • Extending the OLFS • New Dispatch handlers • See Writing Custom DIspatch Handlers
Software Libraries • From OPeNDAP: C++, C, Java • OPeNDAP/Unidata: NetCDF • Other groups: Python
OPeNDAP, DAP and Hyrax • OPeNDAP: Runs the open-source project developing and maintaining DAP. OPeNDAP makes a conscious effort to work with other groups. • DAP: A simple protocol, able to take advantage of HTTP’s wide-spread use, that has proven to work well with a variety of data. • Hyrax: A modular, extensible, multi-protocol server for DAP.
DAP Request Processing • DAP requests are made using URLs • URL: <pathname>.<request>[?<constraint>] • Pathname: names a unique data source • Request: what the client wants • Constraint: optional modification to the request
Request Processing • The OLFS parses the URL into the pathname, request and constraint • It uses its knowledge of the BES commands & capabilities to instruct the BES to build the response that will satisfy the request • It packages the resulting response from the BES (e.g., builds a HTTP ‘text/plain’ response document and inserts the correct content).
DAP Constraints • DAP characterizes data sources as a collection of variables (name-type-value tuples). • Constraints are used to ‘project’ and ‘select’ particular variables with a data source. • Using a constraint, part of one variable may be accessed/read/returned from a data source with many variables. • NB: DAP also uses ‘attributes’ to characterize data sources, but attributes don’t play a direct role in the access of/to binary data values.
BES Handler Interaction • Handlers can: • Build responses directly from data • Use output from other handlers • Handlers respond to commands from the ‘command handler’ • The command handler can be replaced to replace the BES’s default command syntax • Handlers are not limited to DAP
DAP Objects • Metadata • Syntax - essentially variable declarations that describe the names and types of the variables in the data source. Referred to as DDS - Dataset Descriptor Structure. • Use - Information about those variables. Often read from the ‘attributes’ in a file (e.g., HDF4) or from some other source. Referred to as DAS - Dataset Attribute Structure. • Data (referred to as ‘DataDDS’) • Combines DDS for the requested variables along with binary data values. Binary data are encoded in XDR.
DAP Responses • DAP responses are returned by HTTP as ‘documents’ • Metadata - both DAS and DDS use text/plain • Data - uses application/octet-stream • Alternatives: Content of both metadata documents may be returned as a single XML response (text/xml) and Data as a multipart document
Building Responses • Responses to requests are built from the one or more of the basic objects. • Handlers in the BES ‘pass around’ DAP objects, using them as a generalized representation of the data source. • Handlers in the BES that generate responses can effectively be combined with any ‘format’ handler.’
Other Responses • Data may be processed and it’s return form changed (e.g., ‘return as ASCII’) • Metadata may be used to build ‘info pages’ or Web Uis • Other information may be returned (software & protocol version numbers, data source organization, et c.)