Earth Science Data and Information System (ESDIS) Project Update

1. Earth Science Data and Information System (ESDIS) Project Update Karen.Michael@nasa.gov, EOSDIS System Manager AMSR Science Team Meeting Sept 23-24, 2014Huntsville, AL

2. ESDIS Project

3. Data Center SIPSs EOSDIS Facilities Data centers, collocated with centers of science discipline expertise, archive and distribute standard data products produced by Science Investigator-led Processing Systems (SIPSs) ASF DAAC SAR Products Sea Ice, Polar Processes SEDAC Human Interactions in Global Change CDDIS Crustal Dynamics Solid Earth NCAR, U of Col. HIRDLS, MOPITT, SORCE LP DAAC Land Processes & Features GSFC GLAS, MODIS, OMI, OBPG GES DISC Atmos Composition & Dynamics, Global Modeling, Hydrology, Radiance Key NSIDC DAAC Cryosphere, Polar Processes OBPG Ocean Biology & Biogeochemistry JPL MLS, TES ORNL DAAC Biogeochemical Dynamics, EOS Land Validation LAADS/ MODAPS Atmosphere LaRC CERES, SAGE III San Diego ACRIM ECS Sites GHRC Hydrological Cycle & Severe Weather PO.DAAC Ocean Circulation Air-Sea Interactions ASDC Radiation Budget, Clouds, Aerosols, Tropo Chemistry GHRC AMSR-E, LIS, AMSR2

4. EOSDIS Key Metrics EOSDIS Science Data Volume Growth EOSDIS Science Data Products Distribution FY2000 through FY2013

5. REVERB - Data and Service Access Client • Primary web-based client for discovering and ordering cross-discipline data from all of EOS Clearing House (ECHO) metadata holdings. • Allows users, including those without specific knowledge of the data, to search science data holdings, retrieve high-level descriptions of data sets, view browse images, and submit orders via ECHO to appropriate data providers. • Users can submit queries using spatial and temporal criteria and examine search results for relevance using built-in tools. https://earthdata.nasa.gov/reverb

6. REVERB - Data and Service Access Client Free Text Search GCMD Science Keywords The GCMD and ECHO repositories will eventually merge to form the Common Metadata Repository (CMR), a single, shared metadata repository for EOSDIS collection and granule level metadata.

7. LANCE – A Component of EOSDIS • The Land Atmosphere Near-real time Capability for EOS (LANCE) is a component of EOSDIS that generates and distributes near real-time (NRT) products from the following instruments: • Terra: MODIS • Aqua: MODIS, AIRS • Aura: OMI, MLS • AMSR-E (Aqua) – ended 2011 • AMSR2 (GCOM-W1) – future • MISR (Terra) – being added • LANCE Objectives: • To leverage science processing expertise to create high quality NRT products • To provide data to applications community with a maximum latency of 3 hours from observation (The standard, science-quality products are typically available with a latency of 20-48 hours) • To provide data products with high reliability using redundant systems • To provide an umbrella environment with uniform high level requirements to better serve user needs (Consistency, Collaboration, Coordination)

8. LANCE Metrics 2014

9. ESDIS Data Discovery Capabilities • Worldview (https://earthdata.nasa.gov/labs/worldview) • Supports time-critical application areas such as wildfire management, air quality measurements, and weather forecasting. • Allows users to interactively browse satellite imagery in near real-time, add layers, and also provides a time series for comparison. • GIBS (Global Imagery Browse Services) • GIBS are a set of standard services to deliver global, full-resolution browse satellite imagery in a highly responsive manner • Full Resolution mosaicked layers that are chopped into imagery tiles stored at predefined zoom levels • Worldview/GIBS currently consists of: • Routine NRT production of: MODIS (C5), AIRS, MLS, OMI, Ocean Color • AMSR-E historical data in process of being added • Just added 5 Cryospheric Layers in Arctic, Antarctic and Geographic projections from 2002-2011 • : Sea Ice Brightness Temperatures (89H GHz, 89V GHz), Sea Ice Concentration (12km,25km), Snow Depth over Ice (12km) • GIBS/Worldview upcoming plans consist of adding: • Routine NRT production of GPM from PPS • Standard SMAP products • MODIS Collection 6 (Forward + Reprocessed)

10. Continuing Browse Enhancements • GIBS and Worldview • Data granules, used to create the imagery are also searchable and can be downloaded through the Worldview interface.  • Delivery via Standard Services OGC WMTS / TWMS / WMS / KML • Clients can be built to use and display images in GIBS – WorldView is an example

11. All GIBS Product Downloads vs LANCE NRT

12. File Transfer Protocol Testing • Compared file transfer protocol performance across different network environments in ESDIS Network lab and to Alaska • FTP, FTPS, HTTP, SCP, SFTP, HTTPS, bbFTP, GridFTP • Varied file size, packet loss rate, round-trip time to emulate different network environments • Test findings: • Multiple stream file transfer protocols perform best sending large files over networks with high packet loss and long round-trip times (WAN) • bbFTP, GridFTP • Single stream file transfer protocols perform best sending small files over networks with short round-trip times (LAN) • FTP, FTPS, HTTP • Protocols that encrypt the data are consistently the slowest. For best performance, data encryption should be avoided unless it is a requirement. • SCP, SFTP, HTTPS

13. Future File Transfer Protocol Work • Continue working with DAACs and EOSDIS partners choose the best performing protocols for file transfers • Some DAACs have started incorporating multi-stream protocols, bbFTP and GridFTP, for sending large files • Test additional protocols, including Aspera and MMC ftp including tests with JAXA

14. Recent and Upcoming Missions

15. Future Suomi National Polar-orbiting Partnership Plans for NASA SDS • Current architecture • SNPP data is processed by NOAA IDPS • ESDIS manages the Science Data Segment (SDS) of SNPP and was responsible for product evaluation for VIIRS, CrIS, ATMS, OMPS Nadir as well as the production and distribution of CERES, and OMPS Limb • Archive and distribution of SNPP is at NOAA CLASS • Future Architecture • NASA will be producing and distributing it’s own products from all SNPP instruments and potentially future JPSS missions including NRT • Science Investigator-led Processing Systems (SIPS) and Science Teams have recently been selected for the following disciplines: Land, Ocean, Atmosphere, Ozone, Sounder

16. Soil Moisture Active Passive • SMAP Soil Moisture Active Profile • ORR Nov 4-5, 2014 • Launch Dec 15, 2014, or January 29, 2015 • Downlink to Alaska, McMurdo, Svalbard, Wallops (backup) • 15 contacts per day; 135 GB/day • 3 year mission timeline • SMAP data will be processed to L0 by EDOS • SMAP science processing will take place at JPL and GSFC/GMAO • Ingest, archive and distribution of SMAP data will be at ASF for Radar data (L1) and NSIDC for Radiometer data (L1-L4)

17. Preservation • NASA is not a “permanent archive” agency • Must maintain “research archive” for as long as data are used for scientific research and/or transition responsibility to permanent archives • Research archive responsibilities persist well beyond lives of missions • NASA works with USGS and NARA for long-term preservation • NASA has to ensure data and other critical items are preserved and made available to permanent archival agencies • NASA has developed Preservation Content Specifications for Earth Science Data • Categories of Data to be preserved include: • Preflight/Pre-Operations • Science Data Products • Mission Data Calibration • Science Data Product Software • Science Data Product Algorithm Input • Science Data Product Validation • Science Data Software Tools

18. Acronyms