noaa integrated ocean observing system ioos data integration framework n.
Download
Skip this Video
Download Presentation
NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework

Loading in 2 Seconds...

play fullscreen
1 / 43

NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework - PowerPoint PPT Presentation


  • 145 Views
  • Uploaded on

NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework. Richard Crout NOAA NWS National Data Buoy Center GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008. Outline . NOAA’s Integrated Ocean Observing System Data Integrated Framework Challenge

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework' - scout


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
noaa integrated ocean observing system ioos data integration framework

NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework

Richard Crout

NOAA NWS National Data Buoy Center

GCOOS Board of Directors Meeting

Corpus Christi, TX

19 August 2008

outline
Outline
  • NOAA’s Integrated Ocean Observing System
  • Data Integrated Framework
    • Challenge
    • Solution - DIF
    • Goals and Objectives
    • Approach
    • Coordination
    • Web Services
    • Status
    • Validation and Verification
  • Regional Implementation
  • Future Challenges
noaa ioos
NOAA IOOS …
  • Is modernizing the way NOAA collects, shares, and uses ocean information
  • Is increasing data interoperability across NOAA and the regions and efficiency of operations
  • Is expediting access to data for improved decision making
  • To achieve these objectives, the Program focuses its contributions in two main areas:
    • Developing the IOOS Data Integration Framework (NOAA DIF-to-National DMAC)
    • Managing the regional IOOS partnership
noaa ioos program focus
NOAA IOOS Program Focus
  • Standards
    • Develop and maintain a process to document, coordinate, evaluate and accept data standards
  • Regions
    • Invest in Regional Associations efforts that contribute to the definition and implementation of IOOS
  • Data Integration Framework (DIF)
    • Conduct a limited scope implementation of IOOS within NOAA as a proof of principle and to gather lessons to inform the larger IOOS effort
data integration framework dif 5 4 12

NOAAIOOS Data Integration Framework

Systems Engineering &

Standards Development

Hurricane Intensity Model

Temperature

Salinity

Sea Level

Currents

Color

Coastal Inundation Model

Harmful Algal Bloom Model

Data Access & Exchange

Data Standards

Integrated Ecosystem Assessment

Temperature

Salinity

Sea Level

Currents

Color

External sources of 5 Core Variables (consistent with NOAA standards)

Data Integration Framework (DIF)5-4-12

Months 0-12

Month 18

Month 24

Month 36

Test & Evaluation

Benchmarked Product Improvements for Operational Use

Integration of 5 IOOS Core Variables

Integrated Variable Ingest for Select Data Products

NOAA 5 Core Variables

Hurricane Intensity Model

  • Integrated information services for NOAA programs
  • Identify observation gaps
  • Validated enhanced data products
  • NOAA’s Data Integration Framework

Enhanced decision support through:

  • Product Enhancement
  • Test & Evaluation
  • Verification & Validation

Systems Engineering:

Coastal Inundation Model

Harmful Algal Bloom Model

Integrated Ecosystem Assessment

NOAA MISSION OBJECTIVES

REGIONAL COASTAL NEEDS

  • Future State:
  • Regional-coastal data integration for
  • Regional scale data and information products and services
ioos functional areas
IOOS Functional Areas

= Data Discovery, Access, Transport

(encodings, metadata, catalog services, access services, conversion services)

Observing Systems

Data Management and Communications (DMAC)

Modeling and Analysis

= Customer

(decision support tool, analysis tool, generic client)

= Provider

(observing system, data archive)

integration challenge multiple unique independent data requests
Integration Challengemultiple, unique, independent data requests

10 distinct data requests

10 algorithms to convert

to model format

10 different load functions

Data provider changes

have ripple effects

Inefficient

Costly processing

Time consuming

Stovepipe highlight

what is the data integrated framework

(Graphic by i3 Aerospace Technologies Pty Ltd– used with permission)

What is the Data Integrated Framework?

Developing the IOOS Data Integration Framework

  • First spiral of IOOS development
    • Core IOOS project
  • Limited scope
    • DMAC functions
      • Common data-sharing infrastructure
      • Agreed-upon standards
      • Aggregation services
      • Web services
    • Implement at selected provider

& customer sites

    • Focus on several core variables
  • Standardize on small number of services & encodings
    • Adopt, adapt, expand, develop
  • Evaluate in FY 2010
noaa ioos data integrated framework
NOAA IOOS Data Integrated Framework

Goals and Objectives

  • Validate premise: data integration to measure improvements
  • System engineering approach
    • Functional Requirements from identified customers
    • Concept of Operations Document
    • Design Document in progress
  • Architectural principles
    • Service oriented architecture
    • Distributed services operated by data stewards
    • Adopt existing standards and specifications
documentation systems engineering approach
DocumentationSystems Engineering Approach

NOAA IOOS Program

Data Integration Framework (DIF)

Functional Requirements Document

Version 1.0

September 17, 2007

NOAA IOOS Program

Data Integration Framework (DIF)

DIF As-Is Baseline Systems Document

Version 1.0

September 19, 2007

NOAA IOOS Program

Data Integration Framework (DIF)

Concept of

Operations

Version 1.0

April 25, 2008

NOAA IOOS Program

Data Integration Framework (DIF)

Master Project Plan

Version 1.0

September 12, 2007

Master Project

Plan

Version 1.0

9/12/07

Functional

Requirements

Document

Version 1.0

9/17/07

Concept of Operations

Version 1.0

4/25/08

DIF As-Is Baseline Systems Document

Version 1.0

9/19/07

noaa ioos dif
NOAA IOOS DIF

Office of National Environmental Satellite, Data, and Information Services (NESDIS)

CoastWatch

National Climate Data Center(NCDC)

National Ocean Data Center (NODC)

National Geophysical Data Center (NGDC)

National Coastal Data Development Center (NCDDC)

Office of Satellite Data Processing and Distribution (OSDPD)

National Marine Fisheries Service

Southwest Fisheries Science Center (SWFSC)

National Ocean Service

Coastal Services Center (CSC)

National Centers for Coastal Ocean Science (NCCOS)

Center for Operational Oceanographic Products and Services (CO-OPS)

National Weather Service

National Data Buoy Center (NDBC)

National Centers for Environmental Prediction (NCEP)

Oceanic and Atmospheric Research

Pacific Marine Environmental Laboratory (PMEL)

ioos data integration framework

Functional Requirements (completed)

NESDIS/NCDDC NOS/CSC

NWS/NDBC NMFS/SWFSC

NESDIS/NGDC NWS/NCEP

OAR/PMEL NOS/CO-OPS

Data Standards (completed)

NESDIS/NCDDC

NESDIS/NGDC

OAR/PMEL

NOS/CSC

CONOPS (completed)

NESDIS/ORA NMFS/SWFSC

NWS/NDBC NOS/CSDL

NOS/CSC NWS/NCEP

NWS/EMC

Integrated Products Team

Chair – Charles Alexander NOAA IOOS Program

National Weather Service (6)

National Environmental Satellite Data and Information Service (6 + 1)

National Marine Fisheries Service (2)

National Ocean Service (9 + 3)

Oceanic and Atmospheric Research (1 + 1)

Design

NOS

NWS

NESDIS

NOS

NMFS

Harmful Algal Blooms

NOS/NCCOS

NOS/CO-OPS

NWS/NDBC

Coastal Inundation

NWS/TPC

NOS/CO-OPS

Web Services and Data Encodings

Chair - Jeff de La Beaujardiere NOAA IOOS Program

NESDIS (5) NWS (2)

NOS (4) OAR (1)

NMFS (1) RA’s (5)

Image Matters

METADATA

NESDIS/NGDC

NESDIS/NCDDC

IOOS Data Integration Framework

Coordination

slide14

DIF Implementation Schedule

Schema and Integration

Final Report

May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb

2008 2009 2010

OSMC – Observing System Monitoring Center

XML – Entensible Markup Language

DCS – Data Content Standard

WMS – Web Map Standard

10/31/2008

Integrated PORTS

Currents (profile)

CO-OPS

8/18/2008

Integrated Sea Level

Temperature, Salinity,

Winds, & Waves

CO-OPS & NDBC

12/31/2008

PORTS Survey

Currents (profile)

CO-OPS

Global Profile Obs

OSMC

6/12/2008

Integrated Currents

CO-OPS & NDBC

9/25/2008

Integrated Color

CoastWatch

11/30/2008

WMS Services Available

NDBC

6/26/2008

DCS Color

NCDDC

9/30/2008

Integrated HF Radar

NDBC

Global and Coastal Point Obs

OSMC

5/12/2008

Deliver XML Schema

NOAA IOOS Program

May Jun Jul Aug Sep Oct Nov Dec

Schema and Integration

Developing the National IOOS Data Integration Framework

slide15

IOOS Data Integration Framework

Data Sources

Gridded Data Imagery Data

KULA01 CWAO 051800

CREX++

T000101 A008 D09040++

71 917 EUREKA 7598 -08593 00010 18 1998 04 29 23 18

061 019 //// //// 375 0082

0000 400 10137 030 0000 200 10000 030 0001 002 09687 037

0002 002 09366 033 0004 002 08831 037 0005 200 08500 036

0007 002 08013 043 0007 002 07881 047 0008 002 07646 037

0009 002 07442 042 0011 200 07000 031 0012 002 06849 027

0013 002 06710 036 0015 002 06291 029 0022 200 05000 028

0025 002 04557 027 0029 002 04065 024 0029 200 04000 020

0032 002 03626 025 0038 002 03000 020 0040 002 02890 021

0040 002 02829 065 0041 002 02726 105 0043 002 02576 118

0044 200 02500 135 0048 002 02218 165 0049 002 02147 161

0050 002 02104 171 0051 002 02031 153 0051 002 02010 159

0051 200 02000 171 0052 002 01941 188 0054 002 01854 198

0056 002 01744 187 0056 002 01717 194 0057 002 01683 191

0058 002 01640 161 0058 002 01623 159 0059 002 01585 168

0059 002 01576 185 0060 002 01545 197 0061 002 01500 202

In-Situ Data

Need for three services! Open Geospatial Consortium (OSG) provides them.

recommended web services and data encodings
Recommended Web Services and Data Encodings

Data Type

Web Service

Encoding

In-situ data (buoys, piers, towed sensors)

OGC Sensor Observation Service (SOS)

XML based on OGC Observations and Measurements (O&M)

Gridded data (model outputs, satellite)

OpenDAP and/or

OGC Web Coverage Service (WCS)

NetCDF using Climate and Forecast (CF) conventions

Images of data

OGC Web Map Service (WMS)

GeoTIFF, PNG etc.-possibly with standardized styles

OGC: Open Geospatial Consortium

XML: Extensible Markup Language

OPeNDAP: Open-source Project for a Network Data Access Protocol

NetCDF: Network Common Data Format

GeoTIFF: Geospatial Tagged Image File Format

PNG: Portable Network Graphics

xml encoding of in situ data sos getobservation result
XML Encoding of In-Situ Data(SOS GetObservation Result)

XML

Extensible Markup LanguageGeneric method for structuring text data

OGC GMLOpen Geospatial Consortium Geography Markup LanguageXML that can represent any geospatial feature

specializedby

OGC O&MObservations and Measurements ModelGML that describes the act of measuring real-world phenomena and the result of the measurement

specializedby

Data providers currently beta-testing this specification(http://www.csc.noaa.gov/ioos/schema/IOOS-DIF/)

ioos data integration framework1
IOOS Data Integration Framework

Sensor Observation Service

NDBC is implementing SOS on a map interface page

[ http://sdf.ndbc.noaa.gov/]

Data may be retrieved as:

Comma separated values (CSV)

NetCDF

XML

Data available:

Sea level

Currents

Salinity

Sea Temperature

Winds & Waves

slide19

IOOS Data Integration FrameworkSensor Observation Service – GetObservation for ocean current vertical profile

slide20

DIF Data Provider Status

Data Integrated from One Provider

Web Coverage Service

& OPeNDAP

Sensor Observation Service

NDBC

Data Content & Encoding Standards

High-Frequency Radar (HFR) surface currents

NDBC DAC- buoys,

C-MAN, VOS, NOS NERRS, etc.

IOOS DAC- Regional observations

TAO DAC- Tropical Atmosphere Ocean

DART DAC- Deep-Ocean Assessment and Reporting of Tsunamis

NDBC Data Assembly Centers (DACs)

User

Requests

Data Integrated from One Provider

Sensor Observation Service

CO-OPS

Data Content & Encoding Standards

National Water Level Observation Network

PORTS®

Web Coverage Service

& OPeNDAP

Data Integrated from One Provider

CoastWatch

Data Content & Encoding Standards

IOOS Variables: temperature, salinity, currents,water level, winds, waves, ocean color (chlorophyll)

Satellite Ocean Color (Aqua MODIS, Terra MODIS, SeaWiFS)

Expected by end CY2008

additional providers services
Additional Providers & Services

SOS

Single

User

Request

WCS/OPeNDAP

NDBC

SOS

CO-OPS

Registry/Catalog Service

WCS/OPeNDAP

CoastWatch

SOS

Data Archives(NODC?)

Data Integrated from

Multiple Providers

SOS

Region 1

WCS/OPeNDAP

Data

Integration

Service

Region 2

WCS/OPeNDAP

Agency 1

IOOS Variables

+ other variables t.b.d.

SOS

Agency 2

slide22

Verificationand Validation

  • Service Function
    • Valid HTTP response on success and error
    • Server load test
  • Controlling Documents
  • OGC SOS Specification
  • OGC O&M Specification
  • Abstract Data Content Standard
  • XML schema
  • XML dictionaries
  • Feature Type Record Definitions
  • Service Content
    • XML is well-formed (obeys generic XML rules)
    • XML is valid (obeys schema rules)
    • XML matches Data Record Definition
  • Service Operations
    • Service can receive data queries
    • Service can send data queries
  • Data Validation
    • Data returned matches query parameters
    • Data complies with Abstract Data Content Standard
    • Data is easily combined/integrated
      • expressions of geographic position are consistent
      • expressions of temporal position are consistent
      • expressions of measured parameters are consistent
slide23

DIF Service Layers

NOAA

DSTs

COTS/

GIS

General

Public

Other

Customers

NOAA

Regional

Partners

Other

Data

Providers

Target for FY2010

Registry/Catalog

Other

Services

Integration

Translation

Web Map Service

Sensor Observation Service

Access

Services

Web Coverage Service

OpenDAP

pathway to a national dmac

FY

08

FY

09

FY

10

FY

14

Initial Concept

Studies

&

Works

of others

Define capability requirements

High

-

level

Requirement

Use Cases

Conops

Analyze the options and determine the optimal approach to fielding the capability

Functional

&

Technical

Requirements

Resources for

Alternatives

Requirements

Analysis of

Alternatives

Gather Data

(

AoA

)

For Existing

&

Planned

DIF

Development

Effort

Economic

Resources

Analysis

of Preferred

Existing & Planned

Analysis of

Alternative

Approaches

Initiatives

Cost

Capability

Data

Development

Strategy

RFIs

We Are Here

KDP

-

1

Implement

Selected Strategy

Pathway to a National DMAC

Developing the National IOOS Data Integration Framework

noaa ioos dif accomplishments
NOAA IOOS DIF Accomplishments

Developing the IOOS Data Integration Framework

  • 7 IOOS variables integrated by end of 2008
  • Focus on major NOAA data providers for greatest impact
    • Example: 83% of NOAA’s ocean temperature integrated and accessible in initial DIF implementation
  • DIF Integrated Products Team (IPT) developing effective, cross-NOAA integration solution
  • DIF implementation software and protocols will be made available to regional partners
    • NDBC provides 39 Regional temperature observations in August 2008
    • Regions to begin implementation in FY09
noaa ioos data integrated framework1
NOAA IOOS Data Integrated Framework

Future Challenges

  • Very short schedule driven by externals
  • NDBC and CO-OPS are breaking new ground
    • Limited IOOS-specific reference material
    • Simultaneous implementation, testing, revision
    • Sensor metadata descriptions (SensorML) not ready
  • Process needed for working with OGC, SSML, etc. to revise/harmonize/promote in FY09
  • Need for better documentation, tools, converters, clients, reference implementation, test suites
  • Need more customers/users
questions
Questions?
  • http://ioos.noaa.gov
slide31

NOAA Coordination

  • Integrated Products Team

11

Chair, Charles Alexander – NOAA IOOS Program

  • National Weather Service
  • National Env. Satellite Data & Informtn. Service
  • National Marine Fisheries Service
  • National Ocean Service
  • Oceanic and Atmospheric Research

Chris Taylor Walter Smith Hendrick Tolman

Steve Baig Avichal Mehra Bill Burnett

Ken Casey Kent Hughes Lewis McCulloch

Julie Bosch Jennifer Fry Phil Keegstra

Tess Brandon Ken McDonald Paul Digiacomo Ted Haberman

Roy Mendelssohn Jim Sargent Dave Foley

Katie Fisher John Ulmer Daniel Martin

Andrea Hardy Rebecca Love Katie Fisher

Shelley Tomlinson Whitley Saumweber Rich Patchen

John Dandy Josh Pederson Jack Harlan Anne Ball (CSC/Ocean.US)

Steve Hankin Gustavo Goni Derrick Snowden

temperature integration build

% of NOAA Integrated

FY08

83%

100%

FY09 (w/ funding)

Under investigation

Temperature Integration Build
national dmac development path based on noaa nasa dod decision processes

High-Level Functional Requirements

Concept of Operations

Implementation

Decision

(~late FY2010)

National DMAC Development Path(based on NOAA, NASA DoD Decision Processes)

26

Initial Concept Studies & Work of Others

Define requirements

Use Cases

Functional & Technical Requirements

Analyze options and

determine optimal approach

Analysis of Alternatives (AoA)

Industry RFI

Economic Analysis of Preferred Alternative

NOAA Data Management Projects

Data Integration Framework (DIF)

Capability Development Strategy

Analysis of Existing Approaches

NSF Ocean Observing Initiative

Regional Coastal Ocean Observing Systems

Federal SOA projects

slide34

IOOS Data Integration FrameworkWeb Map Service (WMS) for Images of Data

Map is a georeferenced “picture” of data

GetCapabilities operation results in a “table of contents” in standardized format

GetMap operation results in image of data customized according to:

Variables of interest

User-specified bounding box

User-specified time

Image size

File format

PNG, GetTIFF, JPEG, GIF

May contain layers of data

slide35

IOOS Data Integration FrameworkWeb Coverage Service for Gridded Data

Coverage is an array of gridded data values

Simplified viewpoint for this discussion, can be more complex

GetCapabilities operation results in a “table of contents”

GetCoverage operation results in a data file containing header and array(s) of numbers customized for:

Variable of interest

User-specified bounding box

User-specified time

File format

netCDF, HDF, floating-point TIFF

DescribeCoverage operation results in metadata about a specific dataset

slide36

IOOS Data Integration FrameworkSensor Observation Service for in-situ Data

Sampling feature discrete locations of measurements

Point, vertical, or horizontal profile, trajectory

… and time series or collections thereof

GetCapabilities operation results in a “table of contents”

GetObservation operation results in an XML data file containing observation values for desired:

Variables of interest

Bounding box

or named geographic feature of interest

or a single sensor

Time

DescribeSensor operation results in XML providing detailed information about a specific sensor

slide37

IOOS Data Integration FrameworkSensor Observation Service

Sensor Ocean Service (SOS)

One of a family of standards and specifications that constitute the Open Geospatial Consortium (OGC), Inc. Sensor Web Enablement (SWE) activity

An application programming interface (API) for providing sensor and observation data via extensible markup language (XML).

Supports:

GetCapabilities – This function returns metadata about this service

DescribeSensor – The function returns detailed sensor characteristics

GetObservation – This function returns observation data of the specified type for the stations within the specified station and date/time range. A bounding box will be added in the future.

Currently, can get:

Most recent observation

Observation at a specific time

Observation for a time range

slide38

IOOS Data Integration FrameworkSensor Observation Service – GetObservation for ocean current vertical profile

slide39

IOOS Data Integration FrameworkSensor Observation Service – GetObservation for ocean current vertical profile

integration challenge multiple unique independent data requests1
Integration Challengemultiple, unique, independent data requests

10 distinct data requests

10 algorithms to convert

to model format

10 different load functions

Data provider changes

have ripple effects

Inefficient

Costly processing

Time consuming

Stovepipe highlight

integration solution dif content and transport standards services1
Integration SolutionDIF content and transport standards/services

DIF content and transport standards/services

slide42

Documentation:Systems Engineering Approach

NOAA IOOS Program

Data Integration Framework (DIF)

Functional Requirements Document

Version 1.0

September 17, 2007

NOAA IOOS Program

Data Integration Framework (DIF)

DIF As-Is Baseline Systems Document

Version 1.0

September 19, 2007

NOAA IOOS Program

Data Integration Framework (DIF)

Concept of

Operations

Version 1.0

April 25, 2008

NOAA IOOS Program

Data Integration Framework (DIF)

Master Project Plan

Version 1.0

September 12, 2007

Master Project

Plan

Version 1.0

9/12/07

Functional

Requirements

Document

Version 1.0

9/17/07

Concept of Operations

Version 1.0

4/25/08

DIF As-Is Baseline Systems Document

Version 1.0

9/19/07

16

data integration framework dif
Data Integration Framework (DIF)

NOAAIOOS Data Integration Framework

Systems Engineering &

Standards Development

Hurricane Intensity Model

Temperature

Salinity

Sea Level

Currents

Color

Coastal Inundation Model

Harmful Algal Bloom Model

Data Access & Exchange

Data Standards

Integrated Ecosystem Assessment

Temperature

Salinity

Sea Level

Currents

Color

External sources of 5 Core Variables (consistent with NOAA standards)

Months 0-12

Month 18

Month 24

Month 36

Test & Evaluation

Benchmarked Product Improvements for Operational Use

Integration of 5 IOOS Core Variables

Integrated Variable Ingest for Select Data Products

NOAA 5 Core Variables

Hurricane Intensity Model

  • Integrated information services for NOAA programs
  • Identify observation gaps
  • Validated enhanced data products
  • NOAA’s Data Integration Framework

Enhanced decision support through:

  • Product Enhancement
  • Test & Evaluation
  • Verification & Validation

Systems Engineering:

Coastal Inundation Model

Harmful Algal Bloom Model

Integrated Ecosystem Assessment

NOAA MISSION OBJECTIVES

REGIONAL COASTAL NEEDS

  • Future State:
  • Regional-coastal data integration for
  • Regional scale data and information products and services