NOAA Challenges in Environmental Data and Information Management

1. NOAA Challenges in Environmental Data and Information Management Prepared for National Research Council Board on Research Data and Information Deirdre Jones NOAA’s National Weather Service and Deputy Chair, Environmental Data Management Committee November 30, 2010

2. Outline • Background • Recent High Visibility Drivers • Ocean Policy Task Force Recommendations • NOAA Climate Service • Deep Water Horizon Incident • Near Term Plans • Data Management Plan Guidance • Improve Data Documentation • Archive and Architecture ConOps • Data Access • Summary of FY 2011 Activities • Next Steps Prepared by Environmental Data Management Committee for BRDI

3. NOAA’s Environmental Information Management Challenges Background • Data Management Challenges are Changing • No longer just about data volume • Data discovery and integration • Data stewardship and information Broad Scope for Environmental Data Stewardship • ~150 Research & Operational Observing Systems • ~4-5 Petabytes of data/year (~15 Pb total) Prepared by Environmental Data Management Committee for BRDI 3

4. Ocean Policy Task Force • ACTIONS: Data and Tools Theme Team • Develop and implement a comprehensive strategy for identifying and integrating relevant NOAA data sets to support regional Coastal and Marine Spatial planning. • Contribute to the design and creation of the interagency NIMS. • Work with NOAA and partners to assess, enhance and provide access to robust and flexible analytical tools to evaluate alternative ocean use scenarios. 2 9 • ACTIONS: • Develop source to repository data management system. • Adopt and implement common standards for data transport, metadata, data models, and data discovery. Prepared by Environmental Data Management Committee for BRDI 4

5. NOAA Climate Service • NCS Core Capability 1: Observing Systems, Data Stewardship, and Monitoring • NCS users will obtain easy and timelyaccess to the nation’s trusted data and information about the current state of the climate system in context with the past. Prepared by Environmental Data Management Committee for BRDI 5

6. Deep Water Horizon Lessons Learned Clear and consistent data management policy Need for better data documentation (metadata) Overarching response plan Immediate post-incident Data management challenge • Quickly identify applicable / available data Post-incident data management challenges Pre-incident (routine) data management challenges • Releasability of data (organizational vs. technical) • Coordination of collection and distribution • Comparison and utilization of modeling outputs and raw observations • Volume and diversity of data • Collected for one-use; potential for multi-use • Infrastructure needed to support multiple access mechanisms Prepared by Environmental Data Management Committee for BRDI 6

7. FY 2011 EDMC Operating Plan Activities • Develop Data Management Plan Guidance • Improve Data Documentation • Develop Archive Architecture ConOps • Enhance Data Access Activities • Develop Data Sharing Policy for NOAA Grants Prepared by Environmental Data Management Committee for BRDI 7

8. Revision of NOAA-wide Policy (NAO) 212-15 Environmental data will be visible, accessible and independently understandable to users, except where limited by law, regulation, policy or by security requirements. • Maintains NOAA’s policy of “full and open access” to environmental data • Provides mechanism for EDMC to develop procedural directives for more detailed guidance • Presents an end-to-end lifecycle framework for the management of environmental data • Next step: Develop Data Management Plan Guidance • Leveraging work underway by number of NOAA programs (e.g. IOOS, Coral Reef Conservation Program, GOES-R) Prepared by Environmental Data Management Committee for BRDI 8

9. FY 2011 EDMC Operating Plan Activities Improve Data Documentation • The primary focus in FY 2011 will be to establish a metadata standard for NOAA and to provide detailed implementation guidance • NOAA will be encouraging the use and convergence on International Standard Organization (ISO) Metadata Standard • Detailed guidance to be built on NMFS Data Documentation Procedural Directive which is nearing completion • NOAA’s Rolling Deck to Repository (R2R) is another example of end-to-end data documentation Rolling Deck to Repository (R2R) Prepared by Environmental Data Management Committee for BRDI

10. Improve Data DocumentationData Documentation Challenge Transition to ISO standards essential to documenting data for access and exploitation Barriers to this transition Ownership of the responsibility to document data Documentation standards are voluminous - much more than legacy protocols preceding the current standards Data providers, data stewards, and data users all have a stake Boils down to resources--why up front planning is important Exemplary practices in the application of ISO-compliant documentation are starting to emerge from a growing and diverse set of mission- and discipline-specific contexts Exploring formation of a consortium with role to Identify and/or develop exemplary practices in data documentation, especially the application of ISO metadata standards, and Offer training and education opportunities that facilitate and promote the replication and even the advancement of these practices. Expertise needed in three areas: Metadata creation Metadata translation Metadata extension Prepared by Environmental Data Management Committee for BRDI

11. Improving Data DocumentationPrototype Metadata Metric Prepared by Environmental Data Management Committee for BRDI

12. FY 2011 EDMC Operating Plan Activities Develop Archive Architecture ConOps Open Archival Information System Reference Model (OAIS-RM) Functions • NOAA is following the International Standard for information preservation (OAIS-RM) • This standard identifies the functions required to provide long-term preservation • NOAA has multiple organization and systems fulfilling these responsibilities • The Concept of Operations will establish the enterprise process and procedures for determining who does what Prepared by Environmental Data Management Committee for BRDI

13. FY 2011 EDMC Operating Plan Activities Enhance Data Access Activities Client Tools / Applications NOAA Portals / Websites GEOSS Portal Geospatial One-Stop Standard Access Services NOAA Metadata Records Data.gov NOAA’s Environmental Data & Information Infrastructure Prepared by Environmental Data Management Committee for BRDI

14. FY 2011 EDMC Operating Plan Activities Develop Data Sharing Policy for NOAA Grants Prepared by Environmental Data Management Committee for BRDI

15. Summary • NOAA’s revamped Environmental Data Management Committee has full work plan for the next few years • Already leveraging partnerships, e.g. short course in collaboration with ESIP, DM Plan guidelines with IWGDD, etc. • Policy will provide the structure to make EDM truly end to end throughout NOAA Line Offices and Programs from planning through preservation • Outreach and education key to ensuring tangible results • BRDI involvement in any of these areas useful and welcome Prepared by Environmental Data Management Committee for BRDI 15

16. Questions

17. Improving Environmental Data and Information Management in NOAA Prepared for NOAA Leadership by the Environmental Data Management Committee BACKUP SLIDES

18. External Engagement Prepared by Environmental Data Management Committee for BRDI 18

19. FY 2011 Activities Mapped to Lifecycle Components FY 2011 EDMC Activities  = work in progress NAO 212-15 Lifecycle Components Prepared by Environmental Data Management Committee for BRDI 19