Thredds data server
This presentation is the property of its rightful owner.
Sponsored Links
1 / 27

THREDDS Data Server PowerPoint PPT Presentation


  • 102 Views
  • Uploaded on
  • Presentation posted in: General

THREDDS Data Server. John Caron Unidata/UCAR Mar 14, 2005. Outline. THREDDS Catalog Services THREDDS Data Server NetCDF-4 NetCDF-Java 2.2. THREDDS. Thematic Realtime Environmental Distributed Data Services Middleware between data providers and data users

Download Presentation

THREDDS Data Server

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Thredds data server

THREDDS Data Server

John Caron

Unidata/UCAR

Mar 14, 2005


Outline

Outline

  • THREDDS Catalog Services

  • THREDDS Data Server

  • NetCDF-4

  • NetCDF-Java 2.2


Thredds

THREDDS

  • Thematic Realtime Environmental Distributed Data Services

  • Middleware between data providers and data users

  • Enable discovery and use of scientific data

  • Not a data portal

  • XML over HTTP

  • Java Servlets


Thredds catalogs

Catalog.xml

Catalog Generator

Catalog.xml

THREDDS Catalogs

HTTP Server

Catalog.xml

Data Server(s)

DODS, ADDE, FTP, HTTP

Client

Application

Datasets

hostname.edu


Dynamic catalogs data services

Catalog Service

Catalog Generator

Catalog.xml

Query Resolver Service

DQC.xml

Resolver Service

URI

URL

Dynamic Catalogs, data services

HTTP Tomcat Server

Catalog.xml

Client

Application

Data Server(s)

DODS, ADDE, FTP, HTTP

Datasets

hostname.edu


Dataset query capability dqc

Dataset Query Capability (DQC)

  • Dynamic and very large datasets

  • NEXRAD (level 3) from IDD

    • 150 stations

    • 20 products

    • Irregular times

  • The DQC describes the queries that the data server is capable of responding to.


Resolver services

Resolver Services

  • Logical Dataset, eg “latest ETA model run”

  • Dataset with Service type “Resolver”

  • On the client, the URI of the logical dataset is sent to the data server

  • The server finds what is available and returns an actual dataset URL.


Limitations

Limitations

  • Catalogs are very general – “data protocol agnostic”.

  • Automatic metadata extraction needs to “crack” the datasets.

  • Data Servers must be separately installed, maintained, etc


Thredds data server1

THREDDS Data Server

  • Add the data access!

  • 100% Java – single thredds.war

    • requirement, not goal

  • Data input is NetCDF Java 2.2 library

  • Data output:

    • OpenDAP

    • WCS (in progress)

    • FTP, others (future)


Thredds data server2

THREDDS Data Server

HTTP Tomcat Server

Catalog.xml

  • THREDDS Server

  • OPeNDAP

  • WCS

Application

NJ22

library

hostname.edu

Datasets

IDD Data


Thredds opendap server

THREDDS OpenDAP Server

  • Catalogs instead of dods_dir

  • Next version of C++ library will also have THREDDS catalogs

  • Catalogs automatically generated

  • Based on DODS-Java library and the Aggregation Server.

    • Aggregation probably done at nj22 level


Thredds wcs server

THREDDS WCS Server

  • WCS 1.0 specification

  • Proposing to add NetCDF as “blessed” binary file return type

    • OGC interoperability experiment

  • Difficult to find clients to work with

    • Working with CADCorp client

    • Liping Di (George Mason)

    • NASA GSFC ‘Earth Science Gateway’

    • Looking for others


Thredds wcs server1

THREDDS WCS Server

  • Return formats

    • Geotiff floating point

    • NetCDF binary

  • No reprojections for now

  • Must have Grid coordinate system

  • Issues

    • Vertical coordinates

    • Time coordinates

    • Large number of coverages (granularity)


Netcdf 4 netcdf java 2 2

NetCDF-4 NetCDF-Java 2.2


Netcdf 4

NetCDF-4

  • Project funded by NASA to create new version of netCDF using the HDF5 file format.

  • “Extend and merge” netCDF and HDF5:

    • Widespread use and simplicity of netCDF-3

    • Generality and performance of HDF5

  • Specifically, we are funded to create netCDF-4 C library API, using HDF5 library underneath.

  • Russ Rew (PI), Ed Hartnett (Unidata), Mike Folk (co-PI), Quincey Koziol (NCSA)


Netcdf 4 c library

netCDF-3

Interface

netCDF-4 Library

HDF5 Library

NetCDF-4 Architecture

NetCDF-4 C Library


Netcdf 4 and java

NetCDF-4 and Java

  • 100% Java library for netCDF-4 files possible?

  • NetCDF-Java 2.1 already a 100% Java library for netCDF-3 files (and OPeNDAP)

  • NetCDF-Java 2.2: read HDF5 to determine what netCDF-4 data model should be

  • “Common Data Model” emerged for netCDF, HDF5 and OpenDAP.

    • OpenDAP 4.0 protocol being worked on

    • The 3 groups are actively discussing


Netcdf java 2 2 nj22

NetCDF-Java 2.2 (nj22)

  • 100% Java library for netCDF, HDF5, OpenDAP, and more

  • THREDDS integrated

  • Under development

    • Alpha release: Nov 2004

    • Beta release: May 2005

    • Release: summer 2005


Thredds data server

THREDDS

Catalog.xml

Application

Scientific Datatypes

NetCDF-Java version 2.2 architecture

Image

Trajectory

Grid

Station

Radial

NetcdfDataset

NetcdfFile

OpenDAP

ADDE

HDF5

I/O service provider

NetCDF-3

NetCDF-4

GRIB

NIDS

GINI

Nexrad

DSMP


I o service provider

I/O Service Provider

Implement this interface:

public interface IOServiceProvider {

boolean isValidFile( RandomAccessFile raf);

void open( RandomAccessFile raf, NetcdfFile ncfile);

Array readData( Variable v2, List section);

// only if you use Structures

Array readNestedData( Variable v2, List section);

}


Thredds data server

CDM

Goal: N + M instead of N * M things on your TODO List

File Format

#1

Visualization

&Analysis

NetCDF file

File Format

#2

OpenDAP Server

File Format

#N

WCS Service


Thredds data server

Common Data Model (CDM)


Thredds data server

Coordinate Systems

  • Same underlying mathematics as VisAD, ASCII


Scientific data types

Scientific Data Types

  • Point / Station (metars)

    • Unconnected

    • DQCs used with IDD station/radar data

  • Trajectory (floating buoys)

    • 1D connected

  • Radial (NEXRAD)

    • elevation ,azimuth, distance

  • Grid (Model output)

    • cartesian x, y, z

    • WCS


Thredds data server3

THREDDS Data Server

HTTP Tomcat Server

Catalog.xml

  • THREDDS Server

  • OPeNDAP

  • WCS

Application

NJ22 UI library

NJ22

library

NJ22 library

hostname.edu

Datasets

IDD Data


Summary

Summary

  • THREDDS data server combines Catalog Services and data servers

  • As nj22 matures, data server will have more functionality

  • OpenDAP, WCS

  • NetCDF-4: simplicity of netCDF and flexibility of HDF5


  • Login