Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS

1. Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS Roland Schweitzer Weathertop Consulting, LLC Steve Hankin and Ansley Manke NOAA/PMEL

2. Highlights • Server-side analysis • Motivation • LAS as OPeNDAP Client and Server • Evolution of the server implementation • Community “call to action” • Summary

3. Server-side Analysis • In general server-side analysis is a computation made by an OPeNDAP server at the request of a client. • The specification of the computation is transmitted to the server via the OPeNDAP URL.

4. Motivation • We are interested in server-side analysis for use with the Live Access Server (LAS). • Primarily as a way to implement comparisons between data defined on different grids • We want our implementation to leverage the analysis capabilities of legacy applications like Ferret and GrADS. • We want to use our experience running legacy applications (like Ferret) from within a Java runtime environment.

5. The Live Access Server (LAS) A highly configurable Web server designed to provide flexible access to geo-referenced scientific data

6. FerretBackend Service Ferret LocalRDBMS Remote RDBMS netCDF data LAS Architecture LAS Product Server Product Server Metadata(XML) client SQLBackend Service DRDSBackend Service product Local netCDF data metadata DRDS server OPeNDAP server product request XML (REST) back endrequest (SOAP)

7. netCDF data 1 netCDF data 2 Comparing OPeNDAP datasets LAS Product Server Metadata(XML) user FerretBackend Service SQLBackend Service DRDSBackend Service Ferret product Suppose the variables are on different grids? OPeNDAP server OPeNDAP server metadata product request XML (REST) back endrequest (SOAP)

8. LAS as an OPeNDAP Server • Data on grids which are available via LAS are guaranteed to be geo-referenced and at least COARDS compliant. • We can often “repair” (including re-gridding) the data and/or metadata by associating a script of Ferret commands with the data source in the LAS configuration. • Wouldn’t it be nice to make these “repaired” data available via OPeNDAP?

9. The Ferret Data Server • FDS made this possible. • FDS provides an OPeNDAP view of the data being served by LAS and makes any transformations specified by the associated script before serving the data. • FDS also implements server-side analysis (including the ability to pass in external data sources).

10. A GDS Digression • The GrADS Data (DODS) Server is the first implementation of this concept. • In fact, FDS used the Anagram framework upon which GDS is built. • Both GDS and FDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting.

11. FDS Capabilities • FDS took advantage of several characteristics of the underlying engine (Ferret). • New "virtual" data variables can be defined • Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and “virtual” data variables • Only performs calculations when the data are requested • Only calculates the minimal set needed to fulfill the current request

12. FDS Evolution • Keep these advantages and evolve the implementation. • The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. • Once “plugged-in” clients using nj22 have access to the data from this container. • We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests.

13. Example % more data/simple4.jnl use levitus_climatology let/d=levitus_climatology temp_20 =temp[d=levitus_climatology,z=0:20@sum] set var/title="surface heat content"/units="deg C“ temp_20[d=levitus_climatology]

14. Example dncdump -c http://porter.pmel.noaa.gov:8920/thredds/dodsC/mydata/simple4.jnl float TEMP(ZAXLEVITR, YAXLEVITR, XAXLEVITR) ; TEMP:units = "DEG C" ; TEMP:long_name = "TEMPERATURE" ; TEMP:_FillValue = -1.e+10f ; TEMP:missing_value = -1.e+10f ; TEMP:dataset = "levitus_climatology.cdf" ; float TEMP_20(YAXLEVITR, XAXLEVITR) ; TEMP_20:dataset = "levitus_climatology" ; TEMP_20:direction = "IJ" ; TEMP_20:units = "deg C" ; TEMP_20:long_name = "surface heat content" ; TEMP_20:missing_value = "-1.00000E+34" ; TEMP_20:virtual = "true" ;

15. IOSP* Application The low-level part of the NetCDF-Java version 2.2 architecture NetcdfFile I/O service provider NetCDF-3 NIDS isValid NetCDF-4 GRIB open HDF5 GINI readData Nexrad DMSP Ferret Java Runtime GrADS *Stolen directly from John Caron with only this measly acknowledgement.

16. The THREDDS Data Server • TDS is an OPeNDAP server. • nj22 I/O Service Providers can be plugged in to TDS. • The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret).

17. F-TDS and Server-Side Analysis • A DataSource handler can also be plugged in to TDS which allows custom handling of OPeNDAP requests based on the contents of the HTTPServletRequest Object (and by implication the URL). • We built such a DataSource handler which recognizes URL with embedded analysis expressions. • The three groups of input to the server-side analysis were copied from GDS. • The three sets are: • Data sources (e.g. OPeNDAP URLs). • Analysis “commands” which are implementation specific. • A sub-region.

18. An Analysis URL http://machine:port/thredds/dodsC/ _expr_ {dataset1,dataset2,...} {expression1;expression2;...} {region} .URLsuffix?constraint Part of the original GDS specification, not necessary and not often used with FDS

19. FDS-Specific Example http://host.gov:9090/thredds/dodsC/data/coads _expr_ {http://host.gov:9090/thredds/dodsC/data/levitus} {DIF=SST[d=1]-TEMP[d=2,g=SST[d=1]]} .asc?DIF

20. Community Action • Can we define a implementation-indepedent syntax for server-side analysis requests? • Some common operations (averaging, differencing, linear interpolation) with standard name and syntax • Server-specific (native) operations • What mechanism? • An encoded XML string? • We use this technique in LAS with good success • A simple command language?

21. XML Analysis Expression <datasets>     <dataset id=1>"my_local_dset.nc"</dataset>     <dataset id=2>"http://remote_dset.nc"</dataset></datasets><operation>    <name>DIFF</name>    <arg pos=1 dset=1>sst</arg>    <arg pos=2 dset=2>sst</arg></operation>

22. With “native” Operation <datasets>     <dataset id=1>my_local_dset.nc</dataset>     <dataset id=2>http://remote_dset.nc</dataset></datasets><native_operation> LET diff = sst[d=2]-sst[d=1] </native_operation>

23. Summary • Server-side analysis is critical for the future LAS. • NetCDF Java 2.2 and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). • A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers.

24. From COLA Jennifer Adams Brian Doty Joe Wielgosz From Unidata John Caron Ethan Davis Former TMAP Richard Rogers Yonghua Wei Acknowledgments