Archive architecture way forward
This presentation is the property of its rightful owner.
Sponsored Links
1 / 24

Archive Architecture Way Forward PowerPoint PPT Presentation


  • 82 Views
  • Uploaded on
  • Presentation posted in: General

Archive Architecture Way Forward. DAARWG Meeting. Overview. Developing an Archive Architecture for the NOAA Enterprise Current Archive Activities CLASS Update. Developing an Archive Architecture for the NOAA Enterprise . Need for an Enterprise Approach.

Download Presentation

Archive Architecture Way Forward

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Archive architecture way forward

Archive Architecture Way Forward

DAARWG Meeting


Overview

Overview

  • Developing an Archive Architecture for the NOAA Enterprise

  • Current Archive Activities

  • CLASS Update

DAARWG Meeting


Developing an archive architecture for the noaa enterprise

Developing an Archive Architecture for the NOAA Enterprise


Need for an enterprise approach

Need for an Enterprise Approach

Growing Volume & Complexity of Data

Small volume of “other” data sets, but highly complex (i.e., formats, metadata, quality)

Much more of this data is not in the Data Centers, but is distributed across NOAA

Model data is fastest growing by volume, but more ephemeral

DAARWG Meeting


Need for an enterprise approach1

Need for an Enterprise Approach

NOAA Admin Order 212-15, Section 3. Policy (4 Nov 10)

  • Management of NOAA environmental data will be based upon an end-to-end data management lifecycle that includes:

    • Determining what environmental data are required to be preserved for the long term and how preservation will be accomplished

    • Developing and maintaining metadata throughout the environmental data lifecycle that comply with standards

    • Obtaining user requirements and feedback

    • Developing and following data management plans that are coordinated with the appropriate NOAA archive for all observing and data management systems

    • Conducting scientific data stewardship to address data content, access, and user understanding

    • Providing for delivery to the archive and secure storage

    • Providing for data access and dissemination

    • Enabling integration and/or interoperability with other information and products

What to Archive

How to Archive

DAARWG Meeting


Class as enterprise it solution

CLASS as Enterprise IT Solution

Preliminary Level 1 Requirements Document (L1RD), 6 Nov 08

Scope. Enterprise-wide IT systemsupporting long-term, secure storage of and common access to environmental datasets and information stewarded by NOAA’s Archives.

  • Large Data “Campaigns”: Satellites (NPP/JPSS, GOES, POES, DMSP, MetOp, EOS), Radar (NEXRAD), and NCEPModels (including reanalysis)

  • Enterprise Approach

    • Providing common services for development and operation of IT systems supporting NOAA Archives

    • Consolidating legacy archival storage systems to reduce acquisition costs

    • Relieving data producers of responsibility for archival

Yet there is no NOAA Data Management Framework or Archive Architecture to define the enterprise…

DAARWG Meeting


Need for archive conops

Need for Archive ConOps

Common Theme from Independent Review Teams

  • Independent review by Dolly Perkins, NASA; NESDIS/OSD commissioned; Final Report, 11/25/09

    • ”There does not appear to be any strategic plan for CLASS that lays out its vision as an enterprise system.”

  • Independent review by Tim Hall, Aerospace; CLASS Operations and Planning Board (COPB) commissioned; Final Report, 4/1/11

    • “There is no government CONOPS for CLASS as an enterprise system”

    • “The lack of a CONOPS will continue to frustrate efforts to establish a valid, executable set of joint requirements, and CLASS is unlikely to meet its goals and requirements”

    • “The lack of a government CONOPS for CLASS is a root cause of the inadequacy of the architecture as well as of gaps in joint requirements”

    • “There is no enterprise organization including staff and supporting policy to operate an enterprise system across all data centers”

DAARWG Meeting


Clarify guidance and directives

Clarify Guidance and Directives

Need for Archive ConOps

  • “The DAARWG found…that because the scope of CLASS has evolved over time, there is widespread misunderstanding within NOAA of the CLASS role and purpose…a lack of understanding of the roles and responsibilities in creating and using a NOAA archive is handicapping development.”

  • NAO 212-15 updated

  • OCIO Data Architecture

  • EDMC/AAT accelerating development of

    • Archive Architecture

    • Data Management Directives

  • NESDIS DAA requested COPB/CLASS Project finalize L1RD

NOAA

Administrative Order

212-15

OCIO

Data Architecture

NOSC/EDMC

Archive Architecture

Data Management Directives

NESDIS/COPB

Archive ConOps

CLASS Level 1 Requirements

DAARWG Meeting


Enterprise conops and architecture

Enterprise ConOps and Architecture

Aerospace Contract ($625K, Sep 11 – May 12)

  • Support to NOAA Data Center Enterprise Concept of Operations (CONOPS), Architecture, and Data Management Strategy Development

  • Joint EDMC/Data Center project

  • Objectives and Scope:

    • Assist the Government in developing a Concept of Operations (CONOPS) for the NOAA Data Center archival and stewardship architecture target state

    • Capture the baseline (“as is”) architecture, generate a target architecture, and develop a transition roadmap and deployment/communication strategy

    • Assist the Government in developing an enterprise data management strategy based on government and industry best practices, and including policy and governance recommendations

  • Kick-off Meeting: 17 November

DAARWG Meeting


Enterprise conops and architecture1

Enterprise ConOps and Architecture

Expected Outcomes of the Study

  • Data Center ConOps

  • Flow ConOps into required Federal Enterprise Architecture documents for KDP2

  • KDP2 - Solution Alternatives Identification (for CLASS Enterprise release)

  • Following KDP2, adopt Final L1 Requirements Documents

  • Move rapidly to KDP3 and remaining steps for CLASS Enterprise

DAARWG Meeting


Current archive activities

Current Archive Activities


Archive data management framework

Archive Data Management Framework

Inter-Agency (e.g., NWS-CS MOA, in draft) Memorandum of Agreement (MOA)

Producer-Archive (e.g., NCEP-NCDC SLA, in draft) Service Level Agreement (SLA)

Preliminary Phase

  • NAO 212-15, Management of Environmental Data and Information

  • Procedure for Scientific Records Appraisal and Archive Approval (What to Archive)

  • Open Archival Information System Reference Model (OAIS-RM)

  • Producer-Archive Interface Methodology Abstract Standard (PAIMAS)

Initiate Request to Archive

Initial Decision

Requirements

Documents

What to Archive Process

Recommendation

Package

Final Decision

Formal

Definition,

Transfer and

Validation Phases

How to Archive Process

Design

Documents

Preliminary

Agreement

Design Approval

Submission

Agreement (SA)

Test & Implement

Archive

DAARWG Meeting


Maturity matrix data quality

Maturity Matrix & Data Quality

Example for a Satellite Data Set

DAARWG Meeting


Tiers of service

Tiers of Service

Enterprise Development

DAARWG Meeting


Federation of systems

Federation of Systems

Prototype Capability

  • NCDC partnership with RENCI and DataNet Consortium

  • Prototype “system of systems” framework; federation of NOAA data systems with NOAA archive using iRODS (Integrated Rule-Oriented Data System)

  • Connectivity to data systems such as RENCI, ORNL, OOI, and Earth System Grid (ESG)

  • Pilot Project

    • Federate with RENCI to share 70TB of NEXRAD data

    • Utilize highly distributed computing to derive climate-quality precipitation re-analysis, push data products to NCDC archive system (CLASS)

  • Future Plans to Support Climate Assessments

    • Federate with NOS systems via RENCI to integrate data from OOI with climate data at NCDC

    • Federate with GFDL and ESG to integrate climate model data with in situ and satellite data at NCDC

  • Special Interest Group Climate (SIGClimate): Collaborative effort with Goddard

DAARWG Meeting


Ncdc archive architecture s0

NCDC Archive Architecture (S0)

DAARWG Meeting


Class status update

CLASS Status Update


Need for data center ownership

Need for Data Center Ownership

Common Theme from Independent Review Teams

  • Independent review by Dolly Perkins, NASA; NESDIS/OSD commissioned; Final Report, 11/25/09

    • ”The COPB is not yet fully responsible for or held accountable for CLASS operations. Responsibility and Accountability requires having the Authority which has not been conveyed to the COPB.”

    • ”Communication between the Data Centers (individually and via the COPB) and the CLASS project is inadequate and ineffective. No one other than the CLASS project seems to have current understanding of where CLASS development is heading.”

  • Independent review by Tim Hall, Aerospace; CLASS Operations and Planning Board (COPB) commissioned; Final Report, 4/1/11

    • “At the core of data center concerns is lack of ownership and control of CLASS. CLASS has evolved as a system viewed and treated as outside the architecture of the Data Centers”

DAARWG Meeting


Program management transition to ncdc jun 11

Program Management Transition to NCDC, Jun 11

NCDC

NGDC

NODC

COPB

COWG

Support Services

Division

Remote Sensing Applications Division

Budget Execution - 5%

(T. Leary)

Government Task Monitor – 30%

(N. Ritchey)

COR / Proj Engineer + CWIP – 100%

(Jim Goudouros )

CLASS Program Manager – 100%

(Kern Witcher)

Alt COR – 5%

(J. Niemiec )

IT Specialist/Systems Engineer - 100%

(Jay Morris )

Legend

Budget Analyst– 5%

(L. Cholid)

Operations Manager – 100%

(D. Carter)

NCDC

Inventory Mgr – NCDC – 1%

(K. Tarver )

Other NESDIS

OMB Exhibit 300 – 30%

(K. Tarver, acting )

Contractor

Project Specialist – 100%

(K. Tarver , acting)

Supervisory

Reporting

Inventory Spec – NCDC – 10%

(A. Annis, acting )

ISSO – 100%

(Scott Koger)

DAARWG Meeting


Program management transition to ncdc jun 111

Program Management Transition to NCDC, Jun 11

Change in Approach

  • First and foremost – Deliver on our Commitments to our Customers

    • NPP and GOES-R preparations

    • Day to day ingest, archive, and access

  • Adopt ‘Major’ Program Management Best Practices

    • Portions of NASA Procedural Requirements NPR 7120.5 and NPR 7123.1

    • NOAA Draft 2007 Major Program Acquisition

    • Realign work to be more product/project oriented

  • Adopt Federal Enterprise Architecture with all components documented leading to a KDP-2 for CLASS enterprise software build

DAARWG Meeting


Recent accomplishments

Recent Accomplishments

  • Completed all test in preparation for NPP launch

  • Archived

    • Climate Forecast System Reanalysis and Reforecast (CFSRR)

    • Continuously Operating Reference Stations (CORS)

  • NESDIS approved plan to migrate from legacy systems by FY15

DAARWG Meeting


Next steps

Next Steps

  • Release of 5.4.1; implement final version of NEAAT

  • Complete Cloud Computing Study

  • Migrate data from legacy systems (NEXRAD)

DAARWG Meeting


Summary

Summary

  • EDMC and Data Centers are contracting with Aerospace to develop an Archive Architecture

    • DAARWG should be engaged in this effort

  • CLASS has turned a corner and is becoming fully integrated into the Data Centers

DAARWG Meeting


  • Login