OOI CyberInfrastructure: Technology Overview - Hyrax

1. OOI CyberInfrastructure:Technology Overview - Hyrax January 2009 Claudiu Farcas OOI CI Architecture & Design Team UCSD/Calit2

2. Hyrax Overview • OPeNDAP Hyrax is a server for scientific data provisioning (using DAP2 standard) • Uses a modular architecture to support different application-level protocols • Data access using DAP • Catalogs using THREDDS • Browsing using HTML and ASCII • Modules for data access • Different file types • Potential for database and scripting • Modules for commands • Commands provide varying operations for different protocols

3. OLFS BES Data Hyrax High-level Architecture • Two cooperating components: • OpenDAP Lightweight Front-end Server (OLFS) provides DAP, plain ASCII & HTML interfaces • Back-end server (BES) reads data from repositories • Both parts can be customized • Front-end: different network protocols • Back-end: different data formats/systems

4. Hyrax Architecture • Front-end (OLFS): • Runs inside Tomcat servlet container • Authentication & Authorization performed via Tomcat • Provides DAP2 compliant interface • SOAP interface for programmatic access • Extensible via pluggable dispatch handlers • Back-end (BES): • Data resource access. • Extensible (see Patrick’s slides) • Single/Multiple Machine Installations Possible • Communication between OLFS & BES via TCP/IP or UNIX Sockets

5. OLFS BES Java Servlet Engine Unix Daemon Hyrax Architecture BES Commands DAP2 THREDDS HTML XML- encapsulated object Data Store Optional THREDDS catalogs File system with data files, SQL Database, …

6. OLFS DAP Request Processing • DAP requests are made using URLs • URL: <pathname>.<request>[?<constraint>] • Pathname: names a unique data source • Request: what the client wants • Constraint: optional modification to the request • OLFS • parses the URL into the pathname, request and constraint • uses knowledge of the BES commands & capabilities to instruct the BES how to build the response • packages the resulting response from the BES (e.g., builds a HTTP ‘text/plain’ response document and inserts the correct content).

7. Extending the OLFS • Extension ‘modules’ written in Java • Added to a directory within Tomcat • The new modules have complete access to the request information • Both HTTP GET and POST requests • There is some significant processing done before the handler is called: • Conditional GET Requests • Authorization & Authentication

8. BES Architecture BES Framework Network Protocol and Process start/stop activities PPT* Initialization/ Termination DAP2 Access Data Catalogs BES Commands/ XML Documents Commands** NetCDF3 HDF4 FreeForm Data Store Interfaces … *PPT: point to point transport **Some commands are built in, most load at run-time

9. BES Components • Modular architecture in C++ • Modules interact with the BES and can interact with other modules • Modules can add/modify functionality of BES • New modules are loaded at runtime when needed • The command handler can be replaced to replace the BES’s default command syntax • Handlers are not limited to DAP

10. BES Handler Interaction • Response handlers: • Respond to commands from the ‘command handler’ • Know how to create the response object • Do not necessarily fill in the response object, but know how the response should be filled in (by asking appropriate request handlers) • Build & transmit new responses • Request/Format handlers • Register interests and capabilities (i.e., know-how to fill a response) • Read data and provide object content • Handlers can: • Build responses directly from data • Use output from other handlers • Query one or more handlers for fill-in information • Reporters: Record usage information (e.g., logging, statistics) • Aggregators: Build aggregate responses

11. BES Extensibility • New request handlers (data handlers like netcdf, hdf4) • Request/data handlers know how to fill in a response object. (e.g., how to open a netcdf file, access attributes within that file, and add those attributes to the DAS response object) • New response handlers (flat, Tab, Info) • Response handlers know how to create the response object and what needs to happen to get that response object filled in. (e.g., for a DAP request, the DAP response handler knows to create a DAS object and contact the request handler for each container to fill in the DAS object) • New commands (e.g., hello world example) • Container and Definition storage • Aggregation engines (multiple can be installed) • Methods of returning data (return as nc) • Initialization/Termination callbacks • Exception Handlers (registered with the Exception Manager) • Reporters (data access statistics, metrics, logging, etc)

12. c1 csv c2 csv das def def c1 c1 c2 c2 help das das das help csv csv das help BES Example Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ff command get das for def Definitions def as c1, c2 show help Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func

13. BES Built-in Commands • String commands sent from client to server • show help; (version, process, status, keys) • set container … • show containers; • delete container <container_name>; • delete containers; • define … • show definitions; • delete definition <def_name>; • delete definitions; • set context <name> to <value>; • show context;

14. BES DAP Commands • DAPCommandModule adds: • get das for <def_name> [return as type]; • get dds … • get ddx … • get dods … • show catalog [for “node”]; (info) • dap-server modules (www,usage,ascii) add: • get info_page … • get ascii … • get html_form … • Data handlers (nc, ff, csv) don’t add any new commands.

15. Thank you