tribal water quality data management n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Tribal Water Quality Data Management PowerPoint Presentation
Download Presentation
Tribal Water Quality Data Management

Loading in 2 Seconds...

play fullscreen
1 / 45

Tribal Water Quality Data Management - PowerPoint PPT Presentation


  • 119 Views
  • Uploaded on

Tribal Water Quality Data Management. NCAI Exchange Network Tribal Users Meeting April 9, 2008. Overview. Tribal 106 Grants Requirements for Monitoring and Assessment Essential concepts of Water Quality Data Management Understanding WQX Steps to a WQX Submittal Accessing Data

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Tribal Water Quality Data Management' - thiery


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
tribal water quality data management

Tribal Water Quality Data Management

NCAI Exchange Network

Tribal Users Meeting

April 9, 2008

overview
Overview
  • Tribal 106 Grants Requirements for Monitoring and Assessment
  • Essential concepts of Water Quality Data Management
  • Understanding WQX
  • Steps to a WQX Submittal
  • Accessing Data
  • Upcoming Developments
  • How to decide where to begin
tribal 106 guidance requirement
Tribal 106 Guidance Requirement
  • All tribes must submit a TAR (Tribal Assessment Report) to continue eligibility for 106 funds (8-3)
    • Monitoring Strategy
    • Water Quality Assessment
    • STORET-compatible electronic data
slide4

106 Guidance parameters

Fundamental Programs:

- Dissolved Oxygen (DO)

- Temperature

- pH

- Turbidity

Intermediate Programs:

- Nitrogen

- Phosphorus

Mature Programs:

- Pathogens

- Macroinvertebrates

- Physical Habitat indicators

slide5

10 Elements of a Tribal Monitoring Strategy

I. Monitoring Strategy

II. Monitoring Objectives

III. Monitoring Design

IV. Core Indicators

V. Quality Assurance

VI. Data Management

VII. Data Analysis/Assessment

VIII. Reporting

IX. Programmatic Evaluation

X. General Support and Infrastructure

essentials of data management
Essentials of Data Management
  • Monitoring data consists of:
    • Where sample/measurement was collected
    • When it was collected
    • What was collected
    • How it was collected
    • Why it was collected
    • Who collected it
  • It does not take a lot of information to create a good data set.
slide7

How does data management fit in?

The life of a sample begins here

Field Sheets capture data

Labs generate data

Now what to do with all this data?

slide8

First, make sense of what you have

Organize data electronically

  • Use the tools you’re familiar with:
    • Access
    • Excel
    • Other higher-powered database management system (i.e. Oracle)

Have a plan for growth

Not only will your skill grow, but your data volume will grow. Be prepared for it.

Find out what’s already available

slide9

Data are valuable, plan for re-use

Electronic data are more valuable than data in file cabinets

The more data are re-used, the more valuable they become

Shared data are of even higher value

  • Provide for better planning decisions
  • Incentivize collaborative efforts
  • Make the most use of the monitoring $$s being invested
slide10

How EPA Describes Monitoring Data

Three categories of data:

  • Projects (Why data were collected)

- Brief summary of monitoring plan

  • Monitoring Locations (Where data were collected)

- Describe where monitoring takes place

  • Results (When, How and What were collected)

- Measurements of what were monitored

wqx is designed around these concepts

ORGANIZATION

ORG Description

ORG Electronic Address

ORG Phone

ORG Address

ACTIVITY GROUP

Activity Group Name

Activity Group Type

Activity IDs

PROJECT

Project Description

Project Binary Object

MONITORING LOCATION

Monitoring Location Identity

Monitoring Location Geospatial

Monitoring Location Binary Object

MONITORING ACTIVITY

Activity Description

Activity Location

Sample Description

Sample Prep

Subsample Description

Activity Binary Object

RESULT

Result Description

Result Binary Object

Result Analytical Method

Result Lab Info

Result Detection Quantitation Level

Lab Sample Prep

WQX is designed around these concepts
the wqx schema
The WQX Schema
  • The WQX schema is a definition of the structure and format that data must have to be submitted through WQX.
  • Primary components: Organization, Project, Monitoring Location, Monitoring Activity, Samples, and Results
  • Data elements currently accommodate physical, chemical, and fish tissue data
  • Biological monitoring and habitat data flow will be incorporated in the very near future
submitting data using wqx
Submitting data using WQX
  • Water Quality Exchange (WQX) uses the Exchange Network to flow physical, chemical and fish tissue, and soon biological and habitat data from an tribe’s own database to the existing STORET Warehouse
  • Tribes that have been awarded Exchange Network Grants will be able to submit data through WQX
slide14

How WQX Works

Approved XML

Submission

XML Submission

CDX

WQX

Parse and

Load Software

Data Context Error Report

Schema Validation

Receive Error Responses

Data Load Confirmation and Summary

Data Submitter

WQX

ODS

slide16

Steps to Submitting to WQX

  • Review the resources that are available
  • Getting involved with the Exchange Network
  • Getting a NAAS Account
  • Getting an Organization ID
  • Understanding XML (How does my data relate to an XML file?)
  • Submitting a file
  • Understanding errors
  • Understanding the WQX structure
  • Interacting with WQX
slide17

Step 1. Resources Available

  • The STORET/WQX team has several resources available to help you:
    • Available at the STORET web site: www.epa.gov/storet
      • Visit the WQX section: http://www.epa.gov/storet/wqx.html
      • Diagrams and presentations about how WQX works
      • Presentations from others on how they’ve implemented WQX (including tribes)
      • Lists of allowable values for WQX
    • Available at the Exchange Network Site: http://www.exchangenetwork.net/exchanges/water/wqx.htm
    • Read the Flow Configuration Document
      • See example XML Files
      • Examine the Data Exchange Template
    • Last but not least, don’t be afraid to ask for help:
      • Join the STORET list serve: www.epa.gov/storet/listserv.html
      • Call the STORET hot line: 1-800-424-9067
      • end an email to storet@epa.gov
slide18

Step 1. Remember this

All of this information is available at: www.epa.gov/storet

slide19

Step 2. Getting involved in the Exchange Network

  • Ask yourself the following questions:
    • Does my organization have a node?
    • Have we submitted data for another flow?
    • Did we use a node? Node client? Or CDX Web?
    • Who is my Node Administrator? Do we have one?
  • If the answer to these questions is ‘No’ or ‘I don’t know’ visit: www.exchangenetwork.net
slide20

Step 3. Getting a NAAS Account

  • The NAAS or Network Authentication Authorization Service is a user account that you can use to access the Exchange Network. They can be specific for an individual node or for a particular user
  • NAAS accounts are assigned by Node Administrators. If you don’t have a Node Administrator, contact the CDX help desk at: 888-890-1995 Or epacdx@csc.com
slide21

Step 4. Getting an Org ID

  • An Organization ID (Org ID) is a unique identifier assigned to your organization (i.e. YUROK for the Yurok Tribe).
  • Org IDs are assigned by EPA upon the request of the user.
  • You must have an Org ID to be able to submit to WQX
  • To request an Org ID, send an email to storet@epa.gov
    • Provide the NAAS accounts that you want to have access to your Organization
slide22

Step 5. Understanding XML

  • XML can be intimidating at first
  • There are many ways to create XML, some things to do:
    • The sample files and Data Exchange Template are key to understanding how to create a WQX XML file.
    • Find a programmer who can spend 20-40 hours on designing an XML file
    • Have some test data that are representative of the data you’ll be submitting
    • Test the procedures that the programmer developed (plan for an additional 20-40 hrs testing)
    • Start small, it takes time to create an XML file, don’t waste development time waiting for a computer to process a file.

***You should only have to do this once, after that, you’re just running the procedure.

slide23

Step 6. Submitting a File

  • Once you have a NAAS account and an Org ID, and have successfully created an XML file, you’re ready to submit.
  • The original test file should be small, but representative of the data you’ll be submitting
  • Submit to CDX test first
  • Once you get a successful submission, keep track of the transaction ID
  • Send an email to the CDX help desk:…. Requesting access to WQX production. Provide them the transaction ID of the successful submission.
  • Go ahead an begin submitting to production
slide24

Step 7. Understanding Errors

  • There are two types of errors in WQX:

1. Schema Errors: the submitted file does not conform to the WQX Schema. (Report is available in the file ‘Validation Results’ which is an xml file).

2. Processing Errors: the submitted file does not pass all of the WQX Business Rules (Report is available in the file ‘ProcessingReport.zip’)

slide25

Step 8. Understanding the WQX Structure

  • WQX files should be organized as follows:
    • Header (with processing instructions)
      • Organization
        • Organizational Information
        • Projects
        • Monitoring Locations
        • Activities
          • Linkage to Projects
          • Linkage to Monitoring Locations
          • Results
        • Activity Groups
slide26

Step 9. Interacting with WQX

  • WQX is designed to function in a Node-to-Node environment (i.e. you manage your data, let your computer and Node keep EPA in sync).
  • WQX allows Inserts, Updates, and Deletes via a batch process
  • WQX has Outbound services via the Query and Solicit method data visualization and system synchronization purposes.
accessing data
Accessing Data

The National STORET Data Warehouse serves as the national repository for Modern STORET data as well as WQX data. Users may obtain data from the warehouse through the STORET home page : www.epa.gov/storet

Web services providing direct access to warehouse data will also provide enhanced accessibility to the data.

storet warehouse updates
STORET Warehouse Updates
  • The National STORET Data Warehouse enhancements for improved data accessibility
    • New Watershed Summary – Available Now
    • New Web Services – Available Now
  • Please visit the STORET website for more information : www.epa.gov/storet
slide30

Where do Web Services fit?

Data in/out services via WQX

WH services available via Exch. Net.

Data out services through direct access to the WH

slide33

What’s coming next?

  • WQX 2.0 is near completion
    • Adds Biological data and Habitat data to the schema
    • Supports Bio/Habitat metrics and indexes
    • EPA will continue to support WQX 1.0 for the foreseeable future
  • XML Generation Tool
    • Tool that will support organizations that don’t have the ability to create XML
  • Batch Geo-location tool
    • Tie Monitoring Locations to the National Hydrography Dataset
slide34

Dates to Remember

  • WQX Version 1.0 data flow in production now
    • Physical, chemical, microbiological, and fish tissue data only
  • Add Biological/Habitat data to WQX schema
    • Bio/Habitat WQX in production (May 08)
  • Develop Web tool for XML Generation
    • Tool available for general use (September 08)
  • Batch Geo-location Tool
    • Under development available January 2009
what is the xml generation tool
What is the XML Generation Tool?
  • A web-based application that assists a data provider with the creation of a valid submission file for the Water Quality Exchange.
  • The next generation of SIM and WebSIM, redesigned to target WQX.
slide36

Data-sharing for all

  • This tool is meant to allow those who do not have Exchange Network Nodes the ability to share data over the web
  • The tool will be easy to use and accept basic input formats (i.e. Excel Spreadsheets).
  • Data submitted via this tool will be available on the internet within one week of submission.
how the xml generation tool fits in

State/Tribal Database

WQX Database

How the XML Generation Tool fits in
  • XML Generation Tool
  • Import Flat File
  • Apply Defaults & Translations
  • Validate & Correct
  • Export XML
  • CDX Lite
  • Submit
  • Get Status
  • Download
  • CDX Web
  • Online Registration
  • Log In
  • WQX System
  • Load XML File
  • Update WQX Database
  • Create Processing Report
  • State/Tribal System
  • Retrieve Data from Database
  • Export XML
  • State/Tribal Node
  • Submit
  • Get Status
  • Download

CDX Node

slide38
Demo

By the Way, what happened to National WebSIM?

batch geo locating
Batch Geo-Locating
  • New development that EPA is working on
  • Allow for any new stations submitted via WQX to be automatically georeferenced to the NHD+
  • The Georeferenced reach codes would be returned to the data provider, as well an opportunity to QA the georeferenced locations
  • This will use a concept called ‘Rain Drop’ to ensure that points are tied to the correct NHD reach
slide40

If the two points represent Accuracy/Precison issues then note how raindrop results in same indexed location

  • Note how upper point with straight snap would go to wrong catchment
  • Note how the path does trace the topo delineation for water
why this is important
Why this is Important?
  • Allows for monitoring data to be easily linked to other types of data (i.e. Impaired waters, assessments, permits, etc.)
  • Allows for upstream/downstream analysis
  • Greatly expands the usability of monitoring data
batch indexing timeline
Batch Indexing Timeline
  • This capability should be available January 2009
  • Deployed as a service that any data flow could use (not just for WQX)
  • Also provide some Geospatial QA services (i.e. is the lat/long in the county/HUC/state that the user specified)
where to get started
Where to Get Started
  • Figure out what data you have, and what format it is in.
  • Find out where you stand with the exchange network
  • Visit: www.epa.gov/storet
  • Give us a call, we can walk you through any questions.
web resources
Web Resources
  • www.epa.gov/storet
  • www.exchangenetwork.net/exchanges/water/wqx.htm
  • www.epa.gov/storet/wqx.html
question comments
Question/Comments
  • Contact:
    • Dwane Young, EPA (202)566-0616

Young.dwane@epa.gov

    • Kristen Gunthardt, EPA (202)566-1194

gunthardt.kristen@epa.gov

    • STORET Help Desk 800-424-9067

storet@epa.gov

Join the STORET list serve for more information, including monthly conference calls about STORET and WQX: www.epa.gov/storet/listserv.html