1 / 16

An OPeNDAP server for CORIOLIS data centre

An OPeNDAP server for CORIOLIS data centre. Context and functions Architecture and technical description Planned improvements. T. Loubrieu, A. Forest (IFREMER). DAP4COR : An OPeNDAP server for CORIOLIS data centre. CORIOLIS is an operational data centre for ocean in-situ measurements.

tender
Download Presentation

An OPeNDAP server for CORIOLIS data centre

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An OPeNDAP server for CORIOLIS data centre Context and functions Architecture and technical description Planned improvements T. Loubrieu, A. Forest (IFREMER)

  2. DAP4COR : An OPeNDAP server for CORIOLIS data centre • CORIOLIS is an operational data centre for ocean in-situ measurements. • As such CORIOLIS : • Collects measurements for various observation systems (ARGO profilers, measurement from research ships, XBT from Volontary observation ships, moorings, …) • Apply various quality controls on datasets. • Disseminates datasets as asked by users (ocean forecasting systems, ARGO, Carbocean, …) • The datasets are fully managed in an Oracle RDBMS until they are disseminated as netCDF files in ftp directories or requested through a GUI web interface.

  3. DAP4COR : external usage requirements • Open a standard interface on CORIOLIS database : • Enable interactive subsetting of the datasets (even with scripts) • Provide an oceanography oriented, high level, interface so that data management and low level information are hidden to the end-user. • Promote CORIOLIS data centre interoperability • Use DAPPER ocean in-situ vertical profile DAP model. • DAP4COR is not intended to be distributed and installed outside of CORIOLIS data centre.

  4. DAP4COR : internal requirements • Demonstrate a robust and highly capable federating service for dissemination : • Improve reactivity in the development of new dissemination tools. • Facilitate the maintenance of the multiple dissemination tools by making them independant from the low level RDBMS model. • Avoid datasets redundancy (Oracle RDBMS / netCDF files) to ease data update management. • Better resources management by using a unique RDBMS pool of connexion. • Use J2EE architecture (java + tomcat) because of system infrastructure and available skills.

  5. DAP4COR : functions • DAP4COR is Web Application enabling : • OPeNDAP request interpretation for ocean in-situ profile dataset subsetting • OPeNDAP, DAPPER-like response generation from RDBMS profile data. • DAP4COR Dataset configuration (on SQL criteria on the RDBMS native storage). • DAP4COR command line client for saving extraction as NetCDF ARGO profiles file.

  6. DAP4COR : request syntax example

  7. DAP4COR : user’s subsetting criterias available • Platform code • Geographical area • Time • Vertical reference : depth, pressure • Available measurements • Measurement values ARGO profiling float T/S diagram On a small ocean box Dissolved oxygen Profiles for 2 weeks Mediterranean Salinity profiles One month

  8. DAP4COR : dataset configuration • Main data (location, time, measurements) are stored in the same 3 tables. • Ancillary data are distributed into numerous exotics tables (CORIOLIS « expert » domain). • DA4COR Datasets can be configured according to user needs by defining low level SQL criteria on the Oracle DBMS : • Subset of platforms (gliders, elephant seals, moorings array…) • Instrument types (CTD, XBT, XCTD, …) • Quality assesment level (real time, near real time, delayed mode…) • Geographical area (ocean basin, WOCE reference section), by using Oracle Spatial functions (GIS). • … • Thanks to that, a most of the RDBMS model complexity is hidden to the user. • The most relevant datasets are configured on server as SQL pre-defined SQL criteria without data redundancy.

  9. DAP4COR : dataset configuration examples • WOCE reference section, for example AX7 • Mediterannean sea • Measurements from elephant seal • Gliders dataset • GTS datasets (BATHY, TESAC messages) • Sea Surface dataset (for satellite calibration) • Pre-processed profiles : pressure/depth conversion, interpolation on standard vertical levels, parameters conversion…  To Be Done • …

  10. DAP4COR : response format • Ocean in-situ vertical profiles as proposed in PMEL’s DAPPER server : • One sequence of profiles • For each profile : • First level information (time, location + quality control results) • Metadata (profile attributes) • Sequence of the measured levels (vertical reference value, measured parameters values, quality control information) • Maximum number of vertical profiles in each response : 1000

  11. DAP4COR : command line client (dap4cor2netCDF) • Windows/Unix or Linux portable software. • Uses DAP4COR opendap request as input. • Saves the subset as NetCDF ARGO profile file. • Command line tool which can be used as batch (usefull for operational requirements).

  12. DAP4COR : system architecture • Components we rely on : • JDAP 1.1.7 • Tomcat application server (enabling load balancing) • Tomcat JDBC connection pool • Oracle RDBMS • Oracle spatial functions (GIS) JDAP 1.1.7 DAP4COR DAP4COR Datasets configuration tomcat JDBC connection pool tomcat application server Oracle RDBMS Spatial functions

  13. DAP4COR : a 3 layers software architecture DAP request interpretation/data model generation based on java API (JDAP 1.1.7) In-situ Vertical Profile business interface • Implemented in order to manage : • raw profiles • Or pre-processed profiles (confidential information hidden, vertical reference conversion, interpolated on standard vertical levels…)  To Be Done • Design as a « profile reader » so that memory overflow does not occur. : It reads the lower layer / write the upper layer very frequently so that the smallest possible amount of data is kept is that layer memory. JDBC convenient resultSets

  14. DAP4COR : profile business interface • Main methods : • setConstraint() : initializes the OPeNDAP subsetting criterias (dataset, projected parameters, selection parameters) • getStationRoughNumber() – getStationPreciseNumber(): maximum profile quota analysis. • Prepare() : initializes the requested profiles subset (before reading) • Profiles subset reading methods (mapped on DAPPER sequences order) : • nextStation() : move to next vertical profile • getStationElementValue() : get global information on the current vertical profile (time, location, platform, …) • nextLevel() : move to next vertical level for the current profile • nextProfileLevelElementValue() : get and move to next measurement for the current vertical level. • Interface implementation is a parameter of the dataset configuration so that we can define, for example : • A dataset of pre-processed profiles with homogeneous vertical references (pressure/depth converted if necessary) • …

  15. DAP4COR : planned improvements • Enable big dataset request (thanks to OPeNDAP indexed requests ?) • Improve performance especially when only the global profile information (time, location, platform code, …) are requested. • Implement alternative profile interface for profile pre-processing (vertical level conversion, interpolation on standard vertical levels…) • Provide a new OpeNDAP interface for other CORIOLIS datasets : • time series at a fixed location • along track observations • Keep on following the DAPPER observation data model convention and improve compliance with it.

  16. DAP4COR : links • Server URL : http://www.ifremer.fr/dap4cor • Documentation and related tools : ftp://ftp.ifremer.fr/ifremer/coriolis/tools/dap4cor Containing : • FAQ document • Java client which enable saving datasets as NetCDF ARGO profiles file (dap4cor2netcdf). • Matlab client script examples • Python client script example

More Related