tribal water quality data management n.
Skip this Video
Loading SlideShow in 5 Seconds..
Tribal Water Quality Data Management PowerPoint Presentation
Download Presentation
Tribal Water Quality Data Management

Loading in 2 Seconds...

play fullscreen
1 / 45

Tribal Water Quality Data Management - PowerPoint PPT Presentation

  • Uploaded on

Tribal Water Quality Data Management. NCAI Exchange Network Tribal Users Meeting April 9, 2008. Overview. Tribal 106 Grants Requirements for Monitoring and Assessment Essential concepts of Water Quality Data Management Understanding WQX Steps to a WQX Submittal Accessing Data

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Tribal Water Quality Data Management' - thiery

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
tribal water quality data management

Tribal Water Quality Data Management

NCAI Exchange Network

Tribal Users Meeting

April 9, 2008

  • Tribal 106 Grants Requirements for Monitoring and Assessment
  • Essential concepts of Water Quality Data Management
  • Understanding WQX
  • Steps to a WQX Submittal
  • Accessing Data
  • Upcoming Developments
  • How to decide where to begin
tribal 106 guidance requirement
Tribal 106 Guidance Requirement
  • All tribes must submit a TAR (Tribal Assessment Report) to continue eligibility for 106 funds (8-3)
    • Monitoring Strategy
    • Water Quality Assessment
    • STORET-compatible electronic data

106 Guidance parameters

Fundamental Programs:

- Dissolved Oxygen (DO)

- Temperature

- pH

- Turbidity

Intermediate Programs:

- Nitrogen

- Phosphorus

Mature Programs:

- Pathogens

- Macroinvertebrates

- Physical Habitat indicators


10 Elements of a Tribal Monitoring Strategy

I. Monitoring Strategy

II. Monitoring Objectives

III. Monitoring Design

IV. Core Indicators

V. Quality Assurance

VI. Data Management

VII. Data Analysis/Assessment

VIII. Reporting

IX. Programmatic Evaluation

X. General Support and Infrastructure

essentials of data management
Essentials of Data Management
  • Monitoring data consists of:
    • Where sample/measurement was collected
    • When it was collected
    • What was collected
    • How it was collected
    • Why it was collected
    • Who collected it
  • It does not take a lot of information to create a good data set.

How does data management fit in?

The life of a sample begins here

Field Sheets capture data

Labs generate data

Now what to do with all this data?


First, make sense of what you have

Organize data electronically

  • Use the tools you’re familiar with:
    • Access
    • Excel
    • Other higher-powered database management system (i.e. Oracle)

Have a plan for growth

Not only will your skill grow, but your data volume will grow. Be prepared for it.

Find out what’s already available


Data are valuable, plan for re-use

Electronic data are more valuable than data in file cabinets

The more data are re-used, the more valuable they become

Shared data are of even higher value

  • Provide for better planning decisions
  • Incentivize collaborative efforts
  • Make the most use of the monitoring $$s being invested

How EPA Describes Monitoring Data

Three categories of data:

  • Projects (Why data were collected)

- Brief summary of monitoring plan

  • Monitoring Locations (Where data were collected)

- Describe where monitoring takes place

  • Results (When, How and What were collected)

- Measurements of what were monitored

wqx is designed around these concepts


ORG Description

ORG Electronic Address

ORG Phone

ORG Address


Activity Group Name

Activity Group Type

Activity IDs


Project Description

Project Binary Object


Monitoring Location Identity

Monitoring Location Geospatial

Monitoring Location Binary Object


Activity Description

Activity Location

Sample Description

Sample Prep

Subsample Description

Activity Binary Object


Result Description

Result Binary Object

Result Analytical Method

Result Lab Info

Result Detection Quantitation Level

Lab Sample Prep

WQX is designed around these concepts
the wqx schema
The WQX Schema
  • The WQX schema is a definition of the structure and format that data must have to be submitted through WQX.
  • Primary components: Organization, Project, Monitoring Location, Monitoring Activity, Samples, and Results
  • Data elements currently accommodate physical, chemical, and fish tissue data
  • Biological monitoring and habitat data flow will be incorporated in the very near future
submitting data using wqx
Submitting data using WQX
  • Water Quality Exchange (WQX) uses the Exchange Network to flow physical, chemical and fish tissue, and soon biological and habitat data from an tribe’s own database to the existing STORET Warehouse
  • Tribes that have been awarded Exchange Network Grants will be able to submit data through WQX

How WQX Works

Approved XML


XML Submission



Parse and

Load Software

Data Context Error Report

Schema Validation

Receive Error Responses

Data Load Confirmation and Summary

Data Submitter




Steps to Submitting to WQX

  • Review the resources that are available
  • Getting involved with the Exchange Network
  • Getting a NAAS Account
  • Getting an Organization ID
  • Understanding XML (How does my data relate to an XML file?)
  • Submitting a file
  • Understanding errors
  • Understanding the WQX structure
  • Interacting with WQX

Step 1. Resources Available

  • The STORET/WQX team has several resources available to help you:
    • Available at the STORET web site:
      • Visit the WQX section:
      • Diagrams and presentations about how WQX works
      • Presentations from others on how they’ve implemented WQX (including tribes)
      • Lists of allowable values for WQX
    • Available at the Exchange Network Site:
    • Read the Flow Configuration Document
      • See example XML Files
      • Examine the Data Exchange Template
    • Last but not least, don’t be afraid to ask for help:
      • Join the STORET list serve:
      • Call the STORET hot line: 1-800-424-9067
      • end an email to

Step 1. Remember this

All of this information is available at:


Step 2. Getting involved in the Exchange Network

  • Ask yourself the following questions:
    • Does my organization have a node?
    • Have we submitted data for another flow?
    • Did we use a node? Node client? Or CDX Web?
    • Who is my Node Administrator? Do we have one?
  • If the answer to these questions is ‘No’ or ‘I don’t know’ visit:

Step 3. Getting a NAAS Account

  • The NAAS or Network Authentication Authorization Service is a user account that you can use to access the Exchange Network. They can be specific for an individual node or for a particular user
  • NAAS accounts are assigned by Node Administrators. If you don’t have a Node Administrator, contact the CDX help desk at: 888-890-1995 Or

Step 4. Getting an Org ID

  • An Organization ID (Org ID) is a unique identifier assigned to your organization (i.e. YUROK for the Yurok Tribe).
  • Org IDs are assigned by EPA upon the request of the user.
  • You must have an Org ID to be able to submit to WQX
  • To request an Org ID, send an email to
    • Provide the NAAS accounts that you want to have access to your Organization

Step 5. Understanding XML

  • XML can be intimidating at first
  • There are many ways to create XML, some things to do:
    • The sample files and Data Exchange Template are key to understanding how to create a WQX XML file.
    • Find a programmer who can spend 20-40 hours on designing an XML file
    • Have some test data that are representative of the data you’ll be submitting
    • Test the procedures that the programmer developed (plan for an additional 20-40 hrs testing)
    • Start small, it takes time to create an XML file, don’t waste development time waiting for a computer to process a file.

***You should only have to do this once, after that, you’re just running the procedure.


Step 6. Submitting a File

  • Once you have a NAAS account and an Org ID, and have successfully created an XML file, you’re ready to submit.
  • The original test file should be small, but representative of the data you’ll be submitting
  • Submit to CDX test first
  • Once you get a successful submission, keep track of the transaction ID
  • Send an email to the CDX help desk:…. Requesting access to WQX production. Provide them the transaction ID of the successful submission.
  • Go ahead an begin submitting to production

Step 7. Understanding Errors

  • There are two types of errors in WQX:

1. Schema Errors: the submitted file does not conform to the WQX Schema. (Report is available in the file ‘Validation Results’ which is an xml file).

2. Processing Errors: the submitted file does not pass all of the WQX Business Rules (Report is available in the file ‘’)


Step 8. Understanding the WQX Structure

  • WQX files should be organized as follows:
    • Header (with processing instructions)
      • Organization
        • Organizational Information
        • Projects
        • Monitoring Locations
        • Activities
          • Linkage to Projects
          • Linkage to Monitoring Locations
          • Results
        • Activity Groups

Step 9. Interacting with WQX

  • WQX is designed to function in a Node-to-Node environment (i.e. you manage your data, let your computer and Node keep EPA in sync).
  • WQX allows Inserts, Updates, and Deletes via a batch process
  • WQX has Outbound services via the Query and Solicit method data visualization and system synchronization purposes.
accessing data
Accessing Data

The National STORET Data Warehouse serves as the national repository for Modern STORET data as well as WQX data. Users may obtain data from the warehouse through the STORET home page :

Web services providing direct access to warehouse data will also provide enhanced accessibility to the data.

storet warehouse updates
STORET Warehouse Updates
  • The National STORET Data Warehouse enhancements for improved data accessibility
    • New Watershed Summary – Available Now
    • New Web Services – Available Now
  • Please visit the STORET website for more information :

Where do Web Services fit?

Data in/out services via WQX

WH services available via Exch. Net.

Data out services through direct access to the WH


What’s coming next?

  • WQX 2.0 is near completion
    • Adds Biological data and Habitat data to the schema
    • Supports Bio/Habitat metrics and indexes
    • EPA will continue to support WQX 1.0 for the foreseeable future
  • XML Generation Tool
    • Tool that will support organizations that don’t have the ability to create XML
  • Batch Geo-location tool
    • Tie Monitoring Locations to the National Hydrography Dataset

Dates to Remember

  • WQX Version 1.0 data flow in production now
    • Physical, chemical, microbiological, and fish tissue data only
  • Add Biological/Habitat data to WQX schema
    • Bio/Habitat WQX in production (May 08)
  • Develop Web tool for XML Generation
    • Tool available for general use (September 08)
  • Batch Geo-location Tool
    • Under development available January 2009
what is the xml generation tool
What is the XML Generation Tool?
  • A web-based application that assists a data provider with the creation of a valid submission file for the Water Quality Exchange.
  • The next generation of SIM and WebSIM, redesigned to target WQX.

Data-sharing for all

  • This tool is meant to allow those who do not have Exchange Network Nodes the ability to share data over the web
  • The tool will be easy to use and accept basic input formats (i.e. Excel Spreadsheets).
  • Data submitted via this tool will be available on the internet within one week of submission.
how the xml generation tool fits in

State/Tribal Database

WQX Database

How the XML Generation Tool fits in
  • XML Generation Tool
  • Import Flat File
  • Apply Defaults & Translations
  • Validate & Correct
  • Export XML
  • CDX Lite
  • Submit
  • Get Status
  • Download
  • CDX Web
  • Online Registration
  • Log In
  • WQX System
  • Load XML File
  • Update WQX Database
  • Create Processing Report
  • State/Tribal System
  • Retrieve Data from Database
  • Export XML
  • State/Tribal Node
  • Submit
  • Get Status
  • Download

CDX Node


By the Way, what happened to National WebSIM?

batch geo locating
Batch Geo-Locating
  • New development that EPA is working on
  • Allow for any new stations submitted via WQX to be automatically georeferenced to the NHD+
  • The Georeferenced reach codes would be returned to the data provider, as well an opportunity to QA the georeferenced locations
  • This will use a concept called ‘Rain Drop’ to ensure that points are tied to the correct NHD reach

If the two points represent Accuracy/Precison issues then note how raindrop results in same indexed location

  • Note how upper point with straight snap would go to wrong catchment
  • Note how the path does trace the topo delineation for water
why this is important
Why this is Important?
  • Allows for monitoring data to be easily linked to other types of data (i.e. Impaired waters, assessments, permits, etc.)
  • Allows for upstream/downstream analysis
  • Greatly expands the usability of monitoring data
batch indexing timeline
Batch Indexing Timeline
  • This capability should be available January 2009
  • Deployed as a service that any data flow could use (not just for WQX)
  • Also provide some Geospatial QA services (i.e. is the lat/long in the county/HUC/state that the user specified)
where to get started
Where to Get Started
  • Figure out what data you have, and what format it is in.
  • Find out where you stand with the exchange network
  • Visit:
  • Give us a call, we can walk you through any questions.
web resources
Web Resources
question comments
  • Contact:
    • Dwane Young, EPA (202)566-0616

    • Kristen Gunthardt, EPA (202)566-1194

    • STORET Help Desk 800-424-9067

Join the STORET list serve for more information, including monthly conference calls about STORET and WQX: