archive architecture way forward n.
Skip this Video
Download Presentation
Archive Architecture Way Forward

Loading in 2 Seconds...

play fullscreen
1 / 24

Archive Architecture Way Forward - PowerPoint PPT Presentation

  • Uploaded on

Archive Architecture Way Forward. DAARWG Meeting. Overview. Developing an Archive Architecture for the NOAA Enterprise Current Archive Activities CLASS Update. Developing an Archive Architecture for the NOAA Enterprise . Need for an Enterprise Approach.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Archive Architecture Way Forward' - sheryl

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
  • Developing an Archive Architecture for the NOAA Enterprise
  • Current Archive Activities
  • CLASS Update

DAARWG Meeting

need for an enterprise approach
Need for an Enterprise Approach

Growing Volume & Complexity of Data

Small volume of “other” data sets, but highly complex (i.e., formats, metadata, quality)

Much more of this data is not in the Data Centers, but is distributed across NOAA

Model data is fastest growing by volume, but more ephemeral

DAARWG Meeting

need for an enterprise approach1
Need for an Enterprise Approach

NOAA Admin Order 212-15, Section 3. Policy (4 Nov 10)

  • Management of NOAA environmental data will be based upon an end-to-end data management lifecycle that includes:
    • Determining what environmental data are required to be preserved for the long term and how preservation will be accomplished
    • Developing and maintaining metadata throughout the environmental data lifecycle that comply with standards
    • Obtaining user requirements and feedback
    • Developing and following data management plans that are coordinated with the appropriate NOAA archive for all observing and data management systems
    • Conducting scientific data stewardship to address data content, access, and user understanding
    • Providing for delivery to the archive and secure storage
    • Providing for data access and dissemination
    • Enabling integration and/or interoperability with other information and products

What to Archive

How to Archive

DAARWG Meeting

class as enterprise it solution
CLASS as Enterprise IT Solution

Preliminary Level 1 Requirements Document (L1RD), 6 Nov 08

Scope. Enterprise-wide IT systemsupporting long-term, secure storage of and common access to environmental datasets and information stewarded by NOAA’s Archives.

  • Large Data “Campaigns”: Satellites (NPP/JPSS, GOES, POES, DMSP, MetOp, EOS), Radar (NEXRAD), and NCEPModels (including reanalysis)
  • Enterprise Approach
    • Providing common services for development and operation of IT systems supporting NOAA Archives
    • Consolidating legacy archival storage systems to reduce acquisition costs
    • Relieving data producers of responsibility for archival

Yet there is no NOAA Data Management Framework or Archive Architecture to define the enterprise…

DAARWG Meeting

need for archive conops
Need for Archive ConOps

Common Theme from Independent Review Teams

  • Independent review by Dolly Perkins, NASA; NESDIS/OSD commissioned; Final Report, 11/25/09
    • ”There does not appear to be any strategic plan for CLASS that lays out its vision as an enterprise system.”
  • Independent review by Tim Hall, Aerospace; CLASS Operations and Planning Board (COPB) commissioned; Final Report, 4/1/11
    • “There is no government CONOPS for CLASS as an enterprise system”
    • “The lack of a CONOPS will continue to frustrate efforts to establish a valid, executable set of joint requirements, and CLASS is unlikely to meet its goals and requirements”
    • “The lack of a government CONOPS for CLASS is a root cause of the inadequacy of the architecture as well as of gaps in joint requirements”
    • “There is no enterprise organization including staff and supporting policy to operate an enterprise system across all data centers”

DAARWG Meeting

clarify guidance and directives
Clarify Guidance and DirectivesNeed for Archive ConOps
  • “The DAARWG found…that because the scope of CLASS has evolved over time, there is widespread misunderstanding within NOAA of the CLASS role and purpose…a lack of understanding of the roles and responsibilities in creating and using a NOAA archive is handicapping development.”
  • NAO 212-15 updated
  • OCIO Data Architecture
  • EDMC/AAT accelerating development of
    • Archive Architecture
    • Data Management Directives
  • NESDIS DAA requested COPB/CLASS Project finalize L1RD


Administrative Order



Data Architecture


Archive Architecture

Data Management Directives


Archive ConOps

CLASS Level 1 Requirements

DAARWG Meeting

enterprise conops and architecture
Enterprise ConOps and Architecture

Aerospace Contract ($625K, Sep 11 – May 12)

  • Support to NOAA Data Center Enterprise Concept of Operations (CONOPS), Architecture, and Data Management Strategy Development
  • Joint EDMC/Data Center project
  • Objectives and Scope:
    • Assist the Government in developing a Concept of Operations (CONOPS) for the NOAA Data Center archival and stewardship architecture target state
    • Capture the baseline (“as is”) architecture, generate a target architecture, and develop a transition roadmap and deployment/communication strategy
    • Assist the Government in developing an enterprise data management strategy based on government and industry best practices, and including policy and governance recommendations
  • Kick-off Meeting: 17 November

DAARWG Meeting

enterprise conops and architecture1
Enterprise ConOps and Architecture

Expected Outcomes of the Study

  • Data Center ConOps
  • Flow ConOps into required Federal Enterprise Architecture documents for KDP2
  • KDP2 - Solution Alternatives Identification (for CLASS Enterprise release)
  • Following KDP2, adopt Final L1 Requirements Documents
  • Move rapidly to KDP3 and remaining steps for CLASS Enterprise

DAARWG Meeting

archive data management framework
Archive Data Management Framework

Inter-Agency (e.g., NWS-CS MOA, in draft) Memorandum of Agreement (MOA)

Producer-Archive (e.g., NCEP-NCDC SLA, in draft) Service Level Agreement (SLA)

Preliminary Phase

  • NAO 212-15, Management of Environmental Data and Information
  • Procedure for Scientific Records Appraisal and Archive Approval (What to Archive)
  • Open Archival Information System Reference Model (OAIS-RM)
  • Producer-Archive Interface Methodology Abstract Standard (PAIMAS)

Initiate Request to Archive

Initial Decision



What to Archive Process



Final Decision



Transfer and

Validation Phases

How to Archive Process





Design Approval


Agreement (SA)

Test & Implement


DAARWG Meeting

maturity matrix data quality
Maturity Matrix & Data Quality

Example for a Satellite Data Set

DAARWG Meeting

federation of systems
Federation of Systems

Prototype Capability

  • NCDC partnership with RENCI and DataNet Consortium
  • Prototype “system of systems” framework; federation of NOAA data systems with NOAA archive using iRODS (Integrated Rule-Oriented Data System)
  • Connectivity to data systems such as RENCI, ORNL, OOI, and Earth System Grid (ESG)
  • Pilot Project
    • Federate with RENCI to share 70TB of NEXRAD data
    • Utilize highly distributed computing to derive climate-quality precipitation re-analysis, push data products to NCDC archive system (CLASS)
  • Future Plans to Support Climate Assessments
    • Federate with NOS systems via RENCI to integrate data from OOI with climate data at NCDC
    • Federate with GFDL and ESG to integrate climate model data with in situ and satellite data at NCDC
  • Special Interest Group Climate (SIGClimate): Collaborative effort with Goddard

DAARWG Meeting

need for data center ownership
Need for Data Center Ownership

Common Theme from Independent Review Teams

  • Independent review by Dolly Perkins, NASA; NESDIS/OSD commissioned; Final Report, 11/25/09
    • ”The COPB is not yet fully responsible for or held accountable for CLASS operations. Responsibility and Accountability requires having the Authority which has not been conveyed to the COPB.”
    • ”Communication between the Data Centers (individually and via the COPB) and the CLASS project is inadequate and ineffective. No one other than the CLASS project seems to have current understanding of where CLASS development is heading.”
  • Independent review by Tim Hall, Aerospace; CLASS Operations and Planning Board (COPB) commissioned; Final Report, 4/1/11
    • “At the core of data center concerns is lack of ownership and control of CLASS. CLASS has evolved as a system viewed and treated as outside the architecture of the Data Centers”

DAARWG Meeting

program management transition to ncdc jun 11
Program Management Transition to NCDC, Jun 11






Support Services


Remote Sensing Applications Division

Budget Execution - 5%

(T. Leary)

Government Task Monitor – 30%

(N. Ritchey)

COR / Proj Engineer + CWIP – 100%

(Jim Goudouros )

CLASS Program Manager – 100%

(Kern Witcher)

Alt COR – 5%

(J. Niemiec )

IT Specialist/Systems Engineer - 100%

(Jay Morris )


Budget Analyst– 5%

(L. Cholid)

Operations Manager – 100%

(D. Carter)


Inventory Mgr – NCDC – 1%

(K. Tarver )


OMB Exhibit 300 – 30%

(K. Tarver, acting )


Project Specialist – 100%

(K. Tarver , acting)



Inventory Spec – NCDC – 10%

(A. Annis, acting )

ISSO – 100%

(Scott Koger)

DAARWG Meeting

program management transition to ncdc jun 111
Program Management Transition to NCDC, Jun 11

Change in Approach

  • First and foremost – Deliver on our Commitments to our Customers
    • NPP and GOES-R preparations
    • Day to day ingest, archive, and access
  • Adopt ‘Major’ Program Management Best Practices
    • Portions of NASA Procedural Requirements NPR 7120.5 and NPR 7123.1
    • NOAA Draft 2007 Major Program Acquisition
    • Realign work to be more product/project oriented
  • Adopt Federal Enterprise Architecture with all components documented leading to a KDP-2 for CLASS enterprise software build

DAARWG Meeting

recent accomplishments
Recent Accomplishments
  • Completed all test in preparation for NPP launch
  • Archived
    • Climate Forecast System Reanalysis and Reforecast (CFSRR)
    • Continuously Operating Reference Stations (CORS)
  • NESDIS approved plan to migrate from legacy systems by FY15

DAARWG Meeting

next steps
Next Steps
  • Release of 5.4.1; implement final version of NEAAT
  • Complete Cloud Computing Study
  • Migrate data from legacy systems (NEXRAD)

DAARWG Meeting

  • EDMC and Data Centers are contracting with Aerospace to develop an Archive Architecture
    • DAARWG should be engaged in this effort
  • CLASS has turned a corner and is becoming fully integrated into the Data Centers

DAARWG Meeting