Overview of CUAHSI HIS Version 1.1

1. Overview of CUAHSI HIS Version 1.1 David R. Maidment Director, Center for Research in Water Resources University of Texas at Austin CUAHSI Biennial Science and Engineering Colloqium Boulder, CO, July 16, 2008

2. HIS Workshop Overview • Morning: 10AM – 12 noon • Overview of CUAHSI HIS and how to use it • Lunch 12 – 1PM • Afternoon 1PM – 3PM • Building a water data service • Break 3PM – 3:30PM • Afternoon 3:30 – 4:30PM • Discussion of future plans

3. HIS Team and Collaborators • University of Texas at Austin – David Maidment, Tim Whiteaker, Ernest To, Bryan Enslein, Kate Marney • San Diego Supercomputer Center – Ilya Zaslavsky, David Valentine, Tom Whitenack • Utah State University – David Tarboton, Jeff Horsburgh, Kim Schreuders, Justin Berger • Drexel University – Michael Piasecki, Yoori Choi • University of South Carolina – Jon Goodall, Tony Castronova • CUAHSI Program Office – Rick Hooper, David Kirschtel, Conrad Matiuk • WATERS Network – Testbed Data Managers • HIS Standing Committee • USGS– Bob Hirsch, David Briar, Scott McFarlane • NCDC – Rich Baldwin

4. HIS Overview Report • Summarizes the conceptual framework, methodology, and application tools for HIS version 1.1 • Shows how to develop and publish a CUAHSI Water Data Service • Available at: http://his.cuahsi.org/documents/HISOverview.pdf

5. Water Data Water quantity and quality Soil water Rainfall & Snow Modeling Meteorology Remote sensing

6. HIS Desktop • Analytical functions controlled interactively by a human operator to generate information products • Observations Data Model and Tools • Excel, ArcGIS, Matlab, Google Earth • Your model or analysis

7. Services-Oriented Architecture for Water Data • Links geographically distributed information servers through internet • Web Services Description Language (WSDL from W3C) • We designed WaterMLas a web services language for water data • Functions for computer to computer interaction HIS Servers in the WATERS Network HIS Central at San Diego Supercomputer Center Web Services

8. Get Data WaterML HIS Central National Water Metadata Catalog Get Metadata

9. Synthesis and communication of the nation’s water data http://his.cuahsi.org Government Water Data Academic Water Data National Water Metadata Catalog Hydroseek WaterML

10. CUAHSI Point Observation Data Services • Data Loading • Put data into the CUAHSI Observations Data Model • Data Publishing • Provide web services access to the data • Data Indexing • Summarize the data in a centralized cataloging system

11. CUAHSI Point Observation Data Services • Data Loading • Put data into the CUAHSI Observations Data Model • Data Publishing • Provide web services access to the data • Data Indexing • Summarize the data in a centralized cataloging system

12. Data Values – indexed by “What-where-when” Time, T t “When” A data value vi (s,t) “Where” s Space, S Vi “What” Variables, V

13. Data Values Table Time, T t vi (s,t) s Space, S Vi Variables, V

14. Observations Data Model Horsburgh, J. S., D. G. Tarboton, D. R. Maidment and I. Zaslavsky, (2008), "A Relational Model for Environmental and Water Resources Data," Water Resour. Res., 44: W05406, doi:10.1029/2007WR006392.

15. CUAHSI Point Observation Data Services • Data Loading • Put data into the CUAHSI Observations Data Model • Data Publishing • Provide web services access to the data • Data Indexing • Summarize the data in a centralized cataloging system

16. Point Observations Information Model Utah State Univ Data Source Little Bear River Network GetSites Little Bear River at Mendon Rd Sites GetSiteInfo GetVariableInfo Dissolved Oxygen Variables GetValues 9.78 mg/L, 1 October 2007, 5PM Values {Value, Time, Metadata} • A data source operates an observation network • A network is a set of observation sites • A site is a point location where one or more variables are measured • A variable is a property describing the flow or quality of water • A value is an observation of a variable at a particular time • A metadata quantity provides additional information about the value

17. Locations Variable Codes Date Ranges WaterML and WaterOneFlow Penn State Data GetSiteInfo GetVariableInfo GetValues Utah State Data NWIS WaterML Data WaterOneFlow Web Service Data Repositories Client EXTRACT TRANSFORM LOAD WaterML is an XML language for communicating water data WaterOneFlow is a set of web services based on WaterML

18. Set of query functions Returns data in WaterML WaterOneFlow

19. Texas Water Data Services Using CUAHSI technology for state and local data sources (using state funding)

20. Publishing an ODM Water Data Service Texas Instream Flow Program Texas Parks and Wildlife Dept Texas Water Development Board Ingest Data From Different Sources SQL Server Integration Services Transform Data into Uniform Format with SSIS Scripts WaterML Load Newly Formatted Data into ODM Tables in MS SQL/Server Observations Data Model (ODM) TPWD ODM TWDB ODM TIFP ODM Wrap ODM with WaterML Web Services for Online Publication

21. Publishing a Hybrid Water Data Service TCOON Metadata are Transferred from XML to the ODM WaterML TCOON DataValues TCOON METADATA ODM Web Services can both Query the ODM for Metadata and use a Web Scraper for Data Values TCOON Water Data Service Get Values from: Metadata From: ODM Database in Austin TCOON Web Site in Corpus Christi GetSites GetSiteInfo GetVariableInfo http://his.crwr.utexas.edu/tcoonts/tcoon.asmx?WSDL Calling the WSDL Returns Metadata and Data Values as if from the same Database

23. CUAHSI Point Observation Data Services • Data Loading • Put data into the CUAHSI Observations Data Model • Data Publishing • Provide web services access to the data • Data Indexing • Summarize the data in a centralized cataloging system

24. Data Series – Metadata description Time End Date Time, t2 There are C measurements of Variable Vi at Site Sj from time t1 to time t2 Count, C Begin Date Time, t1 Site, Sj Space Variable, Vi Variables

25. Series Catalog Time Sj End Date Time, t2 Vi Count, C Begin Date Time, t1 Site, Sj Space Variable, Vi Variables t1 t2 C

26. CUAHSI National Water Metadata Catalog • Indexes: • 50 observation networks • 1.75 million sites • 8.38 million time series • 342 million data values NWIS STORET TCEQ

27. request return return request NAWQA request return return request NAM-12 request return NWIS request return request return return request NARR Data Searching • Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them Searching each data source separately Michael Piasecki Drexel University

28. NAWQA NWIS NARR HODM Semantic Mediation Searching all data sources collectively GetValues GetValues GetValues GetValues generic request GetValues GetValues Michael Piasecki Drexel University GetValues GetValues

29. Hydroseekhttp://www.hydroseek.org Bora Beran, Drexel Supports search by location and type of data across multiple observation networks including NWIS and Storet

30. HydroTagger Ontology: A hierarchy of concepts Each Variable in your data is connected to a corresponding Concept

31. Water Quality in Moreton Bay, Brisbane, Australia (Jane Hunter)

32. Synthesis and communication of the nation’s water data http://his.cuahsi.org Government Water Data Academic Water Data National Water Metadata Catalog Hydroseek WaterML

33. Accomplishments • Observations Data Model (ODM) is robust; • WaterOneFlow web services provide reliable access to ODM data; • WaterML is a common language for water observations data from academic and government sources • National Water Metadata Catalog is the most comprehensive index of the nation’s water observations presently existing.

34. Limitations • Focus on observations data measured as time series at fixed point locations; • Needs adaptation for moving sensors, transects, one-time data collections and field surveys; • Need to work more on • Coverages for weather, climate and remote sensing • Linking data and models • Linking geographic features Observations Models WaterML Geography Coverages