1 / 46

Some Definitions

OPeNDAP (The Open-source Project for a Network Data Access Protocol) APAC Tutorial October 12, 2007 pfox@opendap.org. Some Definitions. DAP = Data Access Protocol Model used to describe the data; Request syntax and semantics; and Response syntax and semantics. OPeNDAP The software;

alvis
Download Presentation

Some Definitions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OPeNDAP(The Open-source Project for a Network Data Access Protocol) APAC Tutorial October 12, 2007pfox@opendap.org APAC, Perth, WA 20071012

  2. Some Definitions DAP = Data Access Protocol • Model used to describe the data; • Request syntax and semantics; and • Response syntax and semantics. OPeNDAP • The software; • Numerous reference implementations; • Core/libraries and services. OPeNDAP Inc. • OPeNDAP is a 501 c(3) non-profit corporation; • Formed to maintain, evolve and promote the discipline neutral DAP that was the DODS core infrastructure. APAC, Perth, WA 20071012

  3. Some Definitions Syntax • The computer representation of a data object - the data types and structures at the computer level; e.g., • T is a floating point array of 20 by 40 elements. Semantics • The information about the contents of an object; e.g., • T is sea surface temperature in degrees Celsius for a certain region of the Earth. APAC, Perth, WA 20071012

  4. Distributed Oceanographic Data System (DODS) • Conceived in 1993 at a workshop held at URI. • Objectives were: • to facilitate access to PI held data as well as data held in national archives and • to allow the data user to analyze data using the application package with which he or she is the most familiar. • Basic system designed and implemented in 1993-1995 by Gallagher and Flierl with NASA and NOAA funding. • From 1994 to present it has been extended with NASA, NOPP, NSF and NOAA funding. APAC, Perth, WA 20071012

  5. Considerations with regard to the development of OPeNDAP • Many data formats • Many different client types Many data providers • Many different semantic representations of the data • Many different security requirements APAC, Perth, WA 20071012

  6. Broad Vision A world in which a single data access protocol is used for the exchange of data between network based applications regardless of discipline. A layer above TCP/IP providing for syntactic and semantic consistency not available in existing protocols such as FTP. APAC, Perth, WA 20071012

  7. Practical Considerations The broad vision: • Is syntactically achievable, but • Is not semantically achievable, at least not in the near term. APAC, Perth, WA 20071012

  8. OPeNDAP Mission Statement To maintain, evolve and promote a data access protocol (DAP) for the syntactically consistent exchange of data over the network. The DAP should provide syntactic interoperability across disciplines and allow for semantic interoperability within disciplines*****. APAC, Perth, WA 20071012

  9. OPeNDAP Vision Statement To achieve the mission: • OPeNDAP will be Non-profit. • Easier to obtain federal funds. • The DAP is more likely to be adopted. • OPeNDAP software will be open source. • More likely to be adopted. • Need community contributions to software. • OPeNDAP will mix implementation with research. • Implementation - to encourage use. • Research - to keep the protocol current. APAC, Perth, WA 20071012

  10. OPeNDAP Vision Statement(cont) • OPeNDAP will rely primarily on federal funding. • Unlikely to obtain private funding for middleware. • Development to be use case driven. • Aligned with Vision/Mission. • Strategic direction will be sought from an Advisory Board consisting of data system experts and with input from you the community of developers. • OPeNDAP will seek partners. • OPeNDAP will utilize community working groups to develop ‘standards’ related to the DAP, OPeNDAP APAC, Perth, WA 20071012

  11. OPeNDAP tactics • The fundamental objective of OPeNDAP and OPeNDAP Inc. is to facilitate internet access to scientific data • This is done by: • Providing a protocol (DAP) to access data over the internet, • Hiding the format (and organization) in which the data are stored from the user, and • Providing subsetting (and other) capabilities for the data at the server • OPeNDAP is based on a multi-tier architecture • OPeNDAP software is open source • Working groups formed on specific topics APAC, Perth, WA 20071012

  12. OPeNDAP relies on projects • To guide user-based requirements for application needs as well as OPeNDAP core development (use cases) • To provide maintenance and evolution of the core software and documentation • Currently: 7 active projects, covers next ~2-3 years, 2 pending proposals • …… APAC, Perth, WA 20071012

  13. Success What constitutes success of the OPeNDAP mission? Adoption of the DAP across a broad range of disciplines with extensive use in several of these. In order to achieve this the DAP must do the following: • It must be sufficiently flexible, all encompassing, etc. that it can be used across a broad range of data types. • Its implementation must be robust, secure, easy to use, provide for a broad range of services, etc. • The funding stream must be robust. • Active and engaged developer and user community APAC, Perth, WA 20071012

  14. Risks It is still the case that • Some other data access protocols are seen as more attractive regardless of whether or not they are, or • Other data access protocols are developed because their community is not aware of OPeNDAP or of what its capabilities are. • We will compare and contrast some of these today APAC, Perth, WA 20071012

  15. To Succeed OPeNDAP must make sure that: • It’s (server and client) capabilities are well known across a broad range of disciplines. • The data model is inclusive. • The implementation is robust and meets user’s needs. • The DAP coexists with other protocols. • It has a robust funding base. • It has extensive documentation. APAC, Perth, WA 20071012

  16. Robust Funding? • What is an appropriate level of activity (funding) for OPeNDAP? What should OPeNDAP be doing? • Core only? + Clients and servers? + Demonstration projects? Should OPeNDAP be a small staff (core only) or…? What priority should be assigned to the elements currently being developed? Which of these go beyond the core? • DAP4 • AIS - metadata consistency • THREDDS - Catalogs • Toolkits - Matlab, IDL,… • Grid capabilities • OTHERS? APAC, Perth, WA 20071012

  17. Releases/Support • Periodic releases • User services - OPeNDAP (support@opendap.org) • User support - first line of defense • Manages the opendap-dev discussions APAC, Perth, WA 20071012

  18. Binaries Generated There are approximately 80 binaries built on a nightly basis. They are built for the following platforms/operating systems: • Linux • FC4 • FC5 • MacOS-X (universal binaries when possible) • Windows XP, win32 • Java 1.5 (Tomcat 5.5) • IRIX (in four variants), Solaris, AIX, OSF APAC, Perth, WA 20071012

  19. Communication • Website (http://www.opendap.org and test.opendap.org) • SVN - Code repository (http://scm.opendap.org:8090/trac/browser/) • Trac - Task/milestone repository (http://scm.opendap.org:8090/trac/) • Telecons • Management - Weekly • Developers - Weekly on Monday 11am MT • Twiki -> MediaWiki • Management • Developers • Coming soon http://docs.opendap.org • opendap-dev e-mail list - main mechanism for messages APAC, Perth, WA 20071012

  20. OPeNDAP Community Working Groups • Modeled after best practices of IETF, W3, OGC, IEEE, ISO, and others • Working Groups: • Authentication • Security • Server-side Functions • Virtualization (Aggregation) • Server-side processing • Geospatial Interoperability • Hyrax and *DS (TDS, GDS, FDS, etc.) • Semantics • DAPPER • netCDF C++ client • Response types • Metrics • Asynchronous transactions • DAP4 • Relational Database access via DAP • Wiki (http://docs.opendap.org/index.php/Working_Groups) APAC, Perth, WA 20071012

  21. OPeNDAP Community Working Groups • Terms of Reference (abridged) http://docs.opendap.org/index.php/Terms_of_Reference • Each OPeNDAP Working Group is established to apply members' expertise in their focus area to produce specific deliverables or outcomes • Types: Software and Documentation • Minimum 3 people • 6 month time-frame APAC, Perth, WA 20071012

  22. OPeNDAP System Elements The OPeNDAP data access protocol is used by a variety of system elements. • Servers • Processing Servers • Aggregating Servers - OPeNDAP chains • Clients • Ancillary Information Services • Browser Interfaces • Data System Integrators (ODC) APAC, Perth, WA 20071012

  23. Servers • Servers receive requests and provide responses via the DAP. • Servers convert the data from the form in which they are stored to the OPeNDAP data model. • Servers provide for subsetting of the data and more. APAC, Perth, WA 20071012

  24. netCDF DSP Tables Data Data Data Data General netCDF JGOFS DSP OPeNDAP Servers Flat Binary ESML HDF4 HDF5 SQL FITS CDF CEDAR Data Data Data Data Data Data Data HDF5 FITS FreeForm HDF4 JDBC CDF CEDAR APAC, Perth, WA 20071012

  25. ESG FDS GDS DAPPER CODAR pyDAP pyDAP Data Data Data Data Data Data Data netCDF OPeNDAP netCDF OPeNDAP GRIB BUFR OPeNDAP netCDF OPeNDAP CODAR General General TDS TDS Data Data netCDF OPeNDAP netCDF OPeNDAP OPeNDAP Servers (specialized) APAC, Perth, WA 20071012

  26. Servers • Servers may also provide other services • Directory traversal. • Browser-based form to build URL. • ASCII version of data. • Metadata associated with the data. • Server side functions. APAC, Perth, WA 20071012

  27. TDS JGOFS CODAR Data Data Data ESG FDS GDS DAPPER pyDAP netCDF OPeNDAP General CODAR Data Data Data Data Data netCDF OPeNDAP netCDF OPeNDAP GRIB BUFR OPeNDAP netCDF OPeNDAP General OPeNDAP Aggregation Servers APAC, Perth, WA 20071012

  28. The Aggregation Server: An Example Local OPeNDAP HTML, GIF netCDF Data Set DSP Data Set File File File File File File DSP Aggregation Server Matlab Client Matlab APAC, Perth, WA 20071012

  29. OPeNDAP System Elements The OPeNDAP data access protocol is used by a variety of system elements. • Servers • Processing Servers • Aggregating Servers - OPeNDAP chains • Clients • Ancillary Information Services • Browser Interfaces • Data System Integrators (ODC) APAC, Perth, WA 20071012

  30. Clients • Clients make requests and receive responses via the DAP. • Clients convert data from the OPeNDAP data model to the form required in the client application. APAC, Perth, WA 20071012

  31. IDL Client pyDAP pyDAP netCDF Java netCDF C ArcGIS ArcGIS Ferret GrADS IDV VisAD ncBrowse IDL Access Excel OPeNDAP Clients NCL Client Matlab Client NCL Matlab APAC, Perth, WA 20071012

  32. OPeNDAP System Elements The OPeNDAP data access protocol is used by a variety of system elements. • Servers • Processing Servers • Aggregating Servers - OPeNDAP chains • Clients • Ancillary Information Services • Browser Interfaces • Data System Integrators (ODC) APAC, Perth, WA 20071012

  33. Ancillary Information Service • Current capability: Attributes only • Client-side only • Local and remote resources • Local resource databases The AIS enables users to augment the metadata for a data source in a controlled way without requiring write access to the original data. By using the DAP, users are also isolated from data format issues. APAC, Perth, WA 20071012

  34. AIS enhancements • Remote resource databases • AIS server • AIS for variables These enhancements will greatly expand the usefulness of the AIS: Remote resource databases and an AIS server will enable third-party ‘AIS sites’ which may be sponsored by project offices or institutions, et cetera. AIS for variables will enable adding metadata which are stored as ‘data.’ APAC, Perth, WA 20071012

  35. Proposed AIS Server 0 1 Client linked w/DAP Software AIS Server Data Source 3 2 AIS Resource 0. Client requests metadata from the AIS server (which appears no different from any other DAP server). 1. The AIS server gets metadata from data source 2. The AIS server gets matching the AIS resource using the AIS database and merges it into the metadata. 3. The AIS server returns resulting the metadata object. APAC, Perth, WA 20071012

  36. OPeNDAP System Elements The OPeNDAP data access protocol is used by a variety of system elements. • Servers • Processing Servers • Aggregating Servers - OPeNDAP chains • Clients • Ancillary Information Services • Browser Interfaces • Data System Integrators (ODC) APAC, Perth, WA 20071012

  37. OCAPI • A pure OPeNDAP C API (OCAPI) for the client-side • Applications: • DAP-aware ‘commands’ for commercial analysis programs (e.g., IDL) • Scripting tools (e.g., Perl) APAC, Perth, WA 20071012

  38. ODC - a Data System Integrator GCMD GFDL netCDF URI HDF GSFC Binary Matlab NVODS OPeNDAP Access IDL ncBrowse IDV Ferret VisAD GrADS Excel ODC APAC, Perth, WA 20071012

  39. The Data Access Protocol (DAP) • The DAP has been designed to be as general as possible without being constrained to a particular discipline or world view. • The DAP is a discipline neutral data access protocol; it is being used in astronomy, medicine, earth science,… APAC, Perth, WA 20071012

  40. Data Access Protocol (DAP2) - Current • DAP2 currently a NASA/ESE ‘Standard’ • Current server (OPeNDAP 3.x; aka SERVER3) DAP3 • XML responses (implemented) APAC, Perth, WA 20071012

  41. DAP4 - Late 2007 (?) • DAP4 improvements over DAP3: • Additional datatypes • Swath • Blob - GIF, MPEG,… • Additional functionality • Check sum • Modulo • The additional datatypes will enable the DAP to be used in a wider variety of circumstances and are a direct response to users’ requests. APAC, Perth, WA 20071012

  42. OPeNDAP’s Hyrax (‘Server4’) • Uses a modular architecture to support different application-level protocols • Data access using DAP • Catalogs using THREDDS • Browsing using HTML and ASCII • Modules for data access • Different file types • Potential for database and scripting • Modules for commands • Commands provide varying operations for different protocols APAC, Perth, WA 20071012

  43. Hyrax Architecture OLFS BES Data • OPeNDAP Lightweight Front end Server (OLFS) • Receives requests and asks the BES to fill them • Uses Java Servlets • Does not directly ‘touch’ data • Back End Server (BES) • Reads data files, Databases, et c., returns info • May return DAP objects or other data APAC, Perth, WA 20071012

  44. Info output HTML output ASCII output OPeNDAP Lightweight Front end Server GridFTP DAP2 Request Formulation** Request from client HTTP DAP2 DAP2 BES SOAP-DAP THREDDS Response to client APAC, Perth, WA 20071012

  45. BES BES Framework Network Protocol and Process start/stop activities PPT* Initialization/ Termination DAP2 Access Data Catalogs BES Commands/ XML Documents Commands** NetCDF3 HDF4 FreeForm Data Store Interfaces … *PPT is built in (other protocols) **Some commands are built in APAC, Perth, WA 20071012

  46. Today’s Overview • DAP Servers and Services • DAP Clients and Services APAC, Perth, WA 20071012

More Related