Data Infrastructure for Hydrologic Observations

1. Data Infrastructure for Hydrologic Observations Ilya Zaslavsky Spatial Information Systems Lab San Diego Supercomputer Center UCSD http://his.cuahsi.org http://hiscentral.cuahsi.org http://hydroseek.net http://river.sdsc.edu/ucsddash http://wron.net.au/DemosII/Modules/ODMKMLGatway.aspx SSI Global Water Initiative Seminar Series, February 3, 2009

2. Map for the US Build a common window on water data using web services Observation Stations Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System USGS National Water Information System NOAA Climate Reference Network

3. US Map of USGS Observations Alaska Puerto Rico Hawaii Antarctica

4. Different types of nutrients by decade: Available Data Total

5. Some physical properties by decade: Available Data Total

6. Water Data Web Sites

7. NWISWeb site output # agency_cd Agency Code # site_no USGS station number # dv_dt date of daily mean streamflow # dv_va daily mean streamflow value, in cubic-feet per-second # dv_cd daily mean streamflow value qualification code # # Sites in this file include: # USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC # agency_cd site_no dv_dt dv_va dv_cd USGS 02087500 2003-09-01 1190 USGS 02087500 2003-09-02 649 USGS 02087500 2003-09-03 525 USGS 02087500 2003-09-04 486 USGS 02087500 2003-09-05 733 USGS 02087500 2003-09-06 585 USGS 02087500 2003-09-07 485 USGS 02087500 2003-09-08 463 USGS 02087500 2003-09-09 673 USGS 02087500 2003-09-10 517 USGS 02087500 2003-09-11 454 Time series of streamflow at a gaging station

8. Point Observations Information Model USGS Data Source Return network information, and variable information within the network Streamflow gages Network Return site information, including a series catalog of variables measured at a site with their periods of record Neuse River near Clayton, NC Sites ObservationSeries Discharge, stage, start, end (Daily or instantaneous) Return time series of values Values 206 cfs, 13 August 2006 {Value, Time, Qualifier} • A data source operates an observation network • A network is a set of observation sites • A site is a point location where one or more variables are measured • A variable is a property describing the flow or quality of water • An observation series is an array of observations at a given site, for a given variable, with start time and end time • A value is an observation of a variable at a particular time • A qualifier is a symbol that provides additional information about the value

9. Information model challenges… Sites STORET has stations, and measurement points, at various offsets… Site metadata lacking and inconsistent (e.g. 2/3 no HUC info, 1/3 no state/county info); agency site files need to be upgraded to ODM… A groundwater site is different than a stream gauge… Censored values Values have qualifiers, such as “less than”, “censored”, etc. – per value. Sometimes mixed data types.. Units There are multiple renditions of the same units, even within one repository There may be several units for the same parameter code (STORET) Unit multipliers (e.g. NCDC ASOS) Sources STORET requires organization IDs (which collected data for STORET) in addition to site IDs Time stamps: ISO 8601 Many in local times; conversion needed Variable names and measurement methods don’t match E.g. NWIS parameter # 625 is labeled ‘ammonia + organic nitrogen‘, Kjeldahl method is used for determination but not mentioned in parameter description. In STORET this parameter is referred to as Kjeldahl Nitrogen ‘bed sediment’ and ‘suspended sediment’ medium types in NWIS vs. STORET’s ‘sediment’.

10. CUAHSI Observations Data Model http://his.cuahsi.org/odmdatabases.html

11. Water web pages Water web services Information communication Water Markup Language (WaterML) HyperText Markup Language (HTML)

12. WaterML design principles Driven largely by hydrologists; the goal is to capture semantics of hydrologic observations discovery and retrieval Relies to a large extent on the information model as in ODM (Observations Data Model), and terms are aligned as much as possible Several community reviews since 2005 Driven by data served by USGS NWIS, EPA STORET, multiple individual PI-collected observations Is no more than an exchange schema for CUAHSI web services A fairly simple and rigid schema tuned to the current implementation; the least barrier for adoption by hydrologists Conformance with OGC specs not in the initial scope – but working with OGC on this (OGC Discussion Paper 07-041)

13. Set of query functions Returns data in WaterML Water Data Services NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

14. Hydrologic Information System Service Oriented Architecture HIS Lite Servers Test bed HIS Servers Central HIS servers External data providers Global search (Hydroseek) Deployment to test beds Customizable web interface (DASH) Other popular online clients HTML - XML Desktop clients Data publishing HIS CentralRegistry & Harvester WaterOneFlow Web Services, WaterML WSDL - SOAP Ontology ETL services Controlled vocabularies Metadatacatalogs ArcGIS WSDL and ODM registration Matlab IDL, R Ontology tagging (Hydrotagger) MapWindow ODM DataLoader Excel Streaming Data Loading Programming (Fortran, C, VB) ODMTools Modeling (OpenMI) Server config tools

15. Central HIS Data Services Catalog

17. Semantic Tagging of Harvested Variables

18. Hydroseekhttp://www.hydroseek.net Supports search by location and type of data across multiple observation networks including NWIS, Storet, and academic data

19. HIS Deployment National Hydrologic Information Server San Diego Supercomputer Center • 11 WATERS Network test bed projects • 16 ODM instances (some test beds have more than one ODM instance) • Data from 1246 sites, of these, 167 sites are operated by WATERS investigators

21. Against the NIH Syndrome 2006: CUAHSI HIS web services are discussed on the BASINS mailing list as a new way to access hydrologic data. The list is mostly used by hydrologists and developers outside academia; NCDC develops ASOS web services following WaterML 2007: MOU with USGS; USGS is developing WaterML-compliant GetValues service; GLEON uses an early version of ODM to develop their own schema (VEGA); Phoenix LTER is developing ODM (in MySQL) and WaterMLservices (in Java); A Google Earth-based client for CUAHSI web services is developed at CSIRO, Australia; Deployment to 11 hydrologic observatory test beds, + CBEO (CEOP project) 2008: KISTERS develops WaterML-compliant web services over their database, for a client; MapWindow open source GIS develops WaterOneFlow parsers; Florida, Texas and Idaho use ODM and WaterOneFlow web services to provide access to state data repositories; New Jersey is considering the same; Another CEOP project, at UC-Davis, is implementing ODM (in Postgres) and web services (in Java); Stroud Water Research Center; SBRP; Australian WRON; AWI… More, which we don’t know about…

22. Water Quality in Moreton Bay, Brisbane, Australia (Jane Hunter)

23. Summary • Generic method for managing and publishing observational data • Supports many types of point observational data • Overcomes syntactic and semantic heterogeneity using a standard data model and controlled vocabularies • Supports a national network of observatory test beds but can grow! • WaterML is a common language for water observations data from academic and government sources • Point Observations Data from Agencies and Academic Investigators can be consistently communicated using web services • National Water Metadata Catalog is the most comprehensive index of the nation’s water observations presently existing • Join the Water Data Federation!

24. Consortium of Universities for the Advancement of Hydrologic Science, Inc. 122 US Universities as of July 2008 An organization representing more than one hundred United States universities, receives support from the National Science Foundation to develop infrastructure and services for the advancement of hydrologic science and education in the U.S. http://www.cuahsi.org/

25. CUAHSI Hydrologic Information System Models Databases Analysis Goal: Enhance hydrologic science by facilitating user access to more and better data for testing hypotheses and analyzing processes • Advancement of water science is critically dependent on integration of water information • Querying nation’s repository of water data • Linking small integrated research sites (<100 km2) with global and continental models • Integrating data from multiple disciplines to understand controls on hydrologic cycle • It is as important to represent hydrologic environments precisely with data as it is to represent hydrologic processes with equations Water quantity and quality Rainfall & Snow Soil water Meteorology Remote sensing

26. SDSC Spatial Information Systems Lab http://spatial.sdsc.edu/lab/ Research and system development • Services-based spatial information integration infrastructure • Mediation services for spatial data, query processing, map assembly services • Long-term spatial data preservation • Spatial data standards and technologies for online mapping (SVG, WMS/WFS) • Support of spatial data projects at SDSC and beyond In Geosciences (GEON, CUAHSI, CBEO,…) services In Neurosciences (BIRN, CCDB) In regional development (NIEHS SBRP, Katrina) Contact: zaslavsk@sdsc.edu