1 / 77

Ocean PEATE

Ocean PEATE. Fred Patt. NICSE. I&TSE. Ozone. SD3E. Sounder. Atmosphere. Land. Ocean. PSOE. Ocean PEATE Agenda. Science Team Introduction Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule Documentation Issues. NPP/VIIRS Ocean Team.

kadeem
Download Presentation

Ocean PEATE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ocean PEATE Fred Patt

  2. NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Science Team Introduction • Ocean PEATE Design • ODPS Design Overview • Implementation Plan and Schedule • Documentation • Issues

  3. NPP/VIIRS Ocean Team • Chuck McClain – Ocean Color PI • Peter Minnett (U. Miami) – SST PI • Gene Feldman – Ocean PEATE Manager • VIIRS Ocean Science Team • Wayne Esaias • Barney Balch (Bigelow Lab) • Mike Behrenfeld (Oregon State U.) • Janet Campbell (U. New Hampshire) • Bob Evans (U. Miami) • Stephane Maritorena (UCSB) • Norm Nelson (UCSB) • Dave Siegel (UCSB) • Ken Voss (U. Miami) • Menghua Wang (NOAA/NESDIS)

  4. Ocean Team (cont.) • VIIRS Instrument and Calibration Support • Kevin Turpie • Bob Barnes • Gene Eplee • Gerhard Meister • Validation Support • Sean Bailey • Jeremy Werdell • Software Support • Bryan Franz • Joel Gales • Data System • John Wilding • Systems Management • Paul Smith

  5. NASA/GSFC Ocean Biology Processing Group (OBPG) Projects • Sea-viewing Wide Field-of-view Sensor (SeaWiFS) / Orbview-2 – active • Moderate-resolution Imaging Spectroradiometer (MODIS) / Terra and Aqua – active • Coastal Zone Color Scanner (CZCS) / Nimbus-7 – heritage • Ocean Color and Temperature Scanner (OCTS) / ADEOS-I – heritage • Glory data system prototype – 2009 launch • Aquarius / SAC-D – May 2010 launch • VIIRS / NPP – September 2009 launch

  6. NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Science Team Introductions • Ocean PEATE Design • ODPS Design Overview • Implementation Plan and Schedule • Documentation • Issues

  7. Ocean PEATE Design • The NPP Ocean PEATE will be implemented within the framework and facilities of the current NASA Ocean Data Processing System (ODPS) • This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions.

  8. Element Overview • Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the SD3E and ADS/CLASS • Assess the quality of the NPP Ocean EDRs for accomplishing NASA’s climate research requirements • Provide suggested algorithm improvements to the IDPS via the Project Science Office Element (PSOE) • Process selected data subsets in support of Evaluation and Validation activities

  9. Changes since PDR • Established interface with RSMAS (U. Miami) for SST Validation – using MODIS validation for proof of concept. • Expanded and elaborated on xDR evaluation methodologies (as presented at July Peer Review).

  10. RSMAS SeaBASS VOST SD3E CLASS(ADS) Casa-NOSA AncillaryDataProviders I&TSE Ocean ScienceCommunity NICSE PSOE Ocean PEATE Interface Diagram Analysis Results, Proposed Algorithm Updates xDRs, IPs, Ancillary Data Management Direction xDRs, IPs, Ancillary Data (if unavailable from SD3E) OceanPEATE Pre-flight Algorithms, Data, Info Software, Data Alternate Ancillary Data xDR Eval. Results, Algorithm Updates Calibration Updates and Evaluations Interaction In Situ Data In Situ Data Algorithm Updates, Test Requests & Results Matchups

  11. Ocean PEATE External Interfaces (1 of 2) • SDS Science Data Distribution and Depository Element (SD3E) • Provides NRT access to raw data • Primary source of RDRs • Provides selected SDRs and EDRs • SDS Integration and Test System Element (I&TSE) • Build and test updates to operational code in mini-IDPS • Run tests on selected data per request of PEATE • Archive Distribution Segment (ADS) • Primary source for archived data • xDRs, IPs, Ancillary Data, Operational Algorithm/Source Code and Calibration Products • Ancillary Data Providers (ADP) • Provides alternate ancillary data sets (e.g., ozone, meteorological data sets) • CasaNOSA • Serves as the NPP pre-flight repository of Government held data for distribution to Government user teams • Place to acquire pre-launch NPP algorithms and supported data files

  12. Ocean PEATE External Interfaces (2 of 2) • NASA VIIRS Ocean Science Team (VOST) • Coordinate activities with PEATEs and PSOE on xDR and recommended algorithm improvements. Supports Independent Calibration Validation Activities • NPP Instrument Calibration Support Element (NICSE) • Provides alternative calibration LUTs and recommended improvements to calibration algorithms • PEATE provides results of LUT and algorithm tests • Project Science Office Element (PSOE) • Provides management direction • Accepts algorithm update recommendations • SeaBASS/ODPS • Provides Ocean Color in situ data • RSMAS/U. Miami • Provides SST in situ locations • PEATE provides SST EDR matchups • Ocean Science Community • Relies on Ocean PEATE to provide evaluation products and results

  13. Assumptions – Role of I&TSE • The Ocean PEATE will rely upon the SDS-provided I&TSE (mini-IDPS) to serve as the testbed for algorithm evaluation and potential improvements and will not attempt to duplicate efforts by running the operational code within the PEATE environment. • The staff of the I&TSE will have the ability to modify any operational code as per Ocean PEATE specification and run the code within the operational environment against a set of PEATE-specified test data sets. • The I&TSE will receive and install the most current version of the code and executable programs for all operational IDPS software required to produce NASA-evaluated EDRs, starting from RDRs. • The I&TSE will maintain all processing code under configuration control. The PEATEs will have access to the configuration controlled code.

  14. Ocean PEATE Support for IDPS Software • The Ocean PEATE will maintain a working knowledge of the VIIRS SDR code and the EDR code for the Ocean Color and SST EDRs. • The Ocean PEATE will maintain an understanding of the relevant IP characteristics, but will (initially) assume that the IPs are properly generated. • The Ocean PEATE will design changes to the code in the I&TSE for the purpose of algorithm improvement or problem resolution. • The Ocean PEATE will develop appropriate test cases and request runs to verify and evaluate the changes. • The Ocean PEATE will provide recommended changes to the PSOE, including a description of the proposed change, effect on the EDR performance, and the evaluation of the test runs.

  15. IDPS VIIRS Ocean EDR Data Flow Processing Module VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files Ancillary Files DEM NDT

  16. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files Ancillary Files DEM NDT

  17. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation NAAPS TOD MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment DEM VIIRS_GD_12 Bathymetry Granulation NDT VIIRS_GD_13 Temperature Granulation

  18. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation NAAPS TOD MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation VIIRS_LN_06 Active Fires VIIRS_CM_01 Cloud Mask NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment DEM VIIRS_GD_12 Bathymetry Granulation NDT VIIRS_GD_13 Temperature Granulation

  19. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS_SN_02 Ice Quality VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data VIIRS_CL_01 Cloud Optical Properties Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation VIIRS_ST_02 Surface Temp NAAPS TOD VIIRS_SN_03 Ice Concentration MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation VIIRS_LN_06 Active Fires VIIRS_CM_01 Cloud Mask NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_ST_01 Sea Surface Temperature SST EDR VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation VIIRS_AR_01 Aerosol Type ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment VIIRS_OC_01 Ocean Color / Chlorophyll DEM VIIRS_GD_12 Bathymetry Granulation OCC EDR NDT VIIRS_GD_13 Temperature Granulation

  20. ODPS MODIS Ocean Product Data Flow Processing Module MOD_PR01 Level-0 to 1A MODIS Product Dynamic Ancillary Data MODIS Level-0 MODIS Level-1A Static Ancillary Data MODIS 1 km Level-1B MOD_PR03 Geolocation MOD_PR02 Level-1A to 1B MODIS SST Platform ATTEPH Data MODIS Geolocation MSl12 Level-1B to 2 MODIS Ocean Color MET Ancillary Data Ozone Ancillary Files Land/Water Mask

  21. Evaluation vs. Product Level • Level-1 (SDR) Evaluations • Onboard calibration analyses • Vicarious calibration • Level-2 (EDR) Evaluations • Matchup analyses • Residual detector (striping) and scan (RVS) dependence • Level-3 Product Evaluations • Sensor cross-comparisons • Algorithm comparisons • Temporal anomaly evaluations

  22. SDR Example – Vicarious Calibration

  23. EDR Example – Scan and Detector Dependence

  24. Level-3 Example – Zonal Cross-Comparisons

  25. NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Science Team Introductions • Ocean PEATE Design • ODPS Design Overview • Implementation Plan and Schedule • Documentation • Issues

  26. ODPS Design Overview • Fully automated, distributed data system for acquiring, processing, archiving, and distributing scientific data • Highly scalable • Easily adaptable to support multiple concurrent missions • Graphical user interfaces for controlling and monitoring system functions and activity • Non-platform specific

  27. ODPS Design Philosophy • Building-Block approach • Programs are usually small and do one thing well • Programs are less complex and subsequently easy to maintain • Promotes reuse • Programs loosely coupled so testing and production can be done in the same environment • Adopt basic standards • ANSI, POSIX, C9x • Use existing technology when possible • Exit statuses indicate successful or failure conditions

  28. RDBMS Device Manager File Migration and Management Data Acquisition and Ingest VDC/Scheduler Level 3 Scheduler Data Distribution Components and Subsystems

  29. Components and Subsystems (1 of 2) • RDBMS is the primary element that manages all system activity • Generic core databases support system infrastructure and non-mission-specific functions • Mission databases catalogue products and house mission-specific data and procedures • High level of reuse possible for similar missions; e.g., MODIS Aqua/Terra, SeaWiFS, CZCS, and OCTS are all ocean-color missions and have similar product suites and requirements

  30. Components and Subsystems (2 of 2) • Relational Database Management System (RDBMS) supports all of the system components (subsystems) • VDC/Scheduler is the primary controlling module within the system • Other subsystems are independent modules, yet rely on the VDC/Scheduler for some their functions • Archive Device Manager (ADM) • Data acquisition and ingest • Level-3 Scheduler • File migration and management • Data distribution

  31. ODPS COTS and Freeware • Linux OS (CentOS 4.x) • Solaris OS • Sybase RDBMS • Subversion (source code management) • Pro-active DBA • Interactive Data Language (IDL) • Generic Mapping Tool (GMT) • Netpbm (graphic image toolkit) • HDF5 Library • Languages: C, PERL, SQL

  32. ODPS Architecture: Hardware • Processing Servers • Intel-based dual Xeon / AMD-based dual Opteron • 8 GB RAM • Five 72 GB SCSI drives • Storage Servers • Intel-based P4 / AMD-based single Opteron • 1 GB / 2 GB RAM • 1.5 TB IDE RAID 5 (3ware) / 9.6 TB SATA RAID 6 (Areca) • 2 hot spare drives per RAID5 • Database Server • Sun V880 • 8-16 GB RAM • 6-12 70 GB SCSI HDD

  33. Distribution Servers (FTP) 1 processing node 6 storage nodes 25.5 TB Distribution Servers (web)1 large 3 processing nodes 63 storage nodes 605 TB Ingest Servers 2 SeaSpace ground stations 5 storage nodes 11.2 TB Cal/Val & QC Systems Processing Cluster 34 processing nodes 1.5 TB Extreme Networks Black Diamond 6816 Gigabit Ethernet switch Mission Operations Systems Database Server 1 large server 876 GB Development Servers 1 processing node 2 storage nodes 2.4 TB Testing Cluster 13 test nodes 2 TB Backup Servers 1 large server 876 GB 2 storage nodes 11 TB User Desktops Network Support Systems ODPS Current Components

  34. Building 28 Room W220 Computing Facility

  35. Reliability and Redundancy • Critical components (database server, network systems) have full maintenance contracts to ensure rapid response to problems • Multiple-server components (ingest, processing, storage, distribution) have substantial redundancy to maintain full capability; spares maintained for rapid replacement. • Testing nodes are separated from mainstream production components.

  36. Technology Refresh • Hardware technology advances (CPU, storage) are continuously monitored to select new components for evaluation. • Typical upgrade threshold is a doubling in capacity (~18 months). • Two generations of hardware are generally in use. • Candidate components are procured, installed in testing cluster and rigorously evaluated in a production-like environment. • Following successful evaluation, multiple copies are procured, installed, tested and swapped in for older components. • ODPS design allows new components to be rapidly added to resource tables without interrupting system operations. • Performance of new components is closely evaluated following installation in operations environment. • Critical components are run in parallel with existing system to ensure reliability under production loading.

  37. NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Science Team Introductions • Ocean PEATE Design • ODPS Design Overview • Implementation Plan and Schedule • Documentation • Issues

  38. Ocean PEATE Gap Analysis (1 of 2) • Acquire, ingest and catalog NPP VIIRS data products: RDRs, SDRs and Ocean EDRs (Data Acquisition & Ingest, Device Manager and File Migration and Management). • Status: Development of acquisition and ingest scripts underway based on sample products in SD3E. • Process selected Ocean EDRs (SST and OCC) to Level-3 to support data product and algorithm evaluations (Level-3 Scheduler, VDC and Level-3 binner). • Status: Prototype Level-3 processing has been demonstrated using sample IDPS Build 1.4 OCC and SST EDRs. • Perform VIIRS OCC EDR matchups with SeaBASS Ocean Color in situ data (extract code). • Status: Pending completion of acquisition and ingest capabilities.

  39. Ocean PEATE Gap Analysis (2 of 2) • Incorporate VIIRS SDR processing for vicarious calibration analysis. (*) • Status: Pending availability of sample SDRs in Build 1.5 format • Produce VIIRS proxy data using VOST-developed software (VDC/Scheduler). • Status: Prototype geolocation and EDR simulation developed to test Level-3 binning of EDRs. • Acquire SST in situ data from RSMAS and perform matchups with SST EDRs (*) • Status: Pending final MOU • Support browse and distribution of data products for team members (Data Distribution). • Status: Pending completion of acquisition and ingest capabilities. (*) New PEATE capability identified since PDR

  40. Prototype Level-3 Processing • EDR to Level-3 processing prototype completed using IDPS Build 1.4 EDRs.

  41. Existing Software Reuse • ODPS Components • Database • VDC/Scheduler • Data Acquisition and Ingest • Level-3 Scheduler • File migration and management • Archive Device Manager • Data distribution • Level-2 multi-mission software (vicarious calibration) • Level-3 multi-mission software (long-term trends and comparisons • Level-2 to Level-3 comparison software (residual sensor errors) • SeaBASS (in situ data management) • Matchup/extraction software

  42. Basis of Estimate (1 of 2) • Acquire, ingest and catalog NPP VIIRS data products: • 200 LOC (combined UNIX shell, SQL, C) per new product, for each of the four VIIRS products • Process Ocean EDRs (SST and OCC) to Level-3: • 300 C LOC to add VIIRS Ocean EDR input to existing binning software (L2bin) • 300 SQL LOC per temporal range (10 ranges total) • 200 shell LOC for all ranges • Perform VIIRS EDR matchups: • 100 C LOC + 10 PERL LOC to add VIIRS Ocean EDR input to existing extraction software

  43. Basis of Estimate (2 of 2) • Process SDRs for vicarious calibration analysis: • 1000 C LOC to add VIIRS SDR input to existing Level-2 processing software (MSl12) • Produce VIIRS proxy data: • 100 shell LOC for each processing stage • Support browse and distribution of data products: • 3000 C LOC to implement browse image generation • 100 PERL LOC to add VIIRS products to existing browse and order web site

  44. Ocean PEATE Data Storage Estimate • Assumptions: • Long-term storage is sized for 100% of RDRs and 10% of SDRs and EDRs • SDR volume includes geolocation

  45. New Hardware for the Ocean PEATE • 7 Storage Servers @ 9.6 TB – first-year VIIRS data storage • Additional servers acquired post-launch to handle years 2 – 5 • No new processing or network capacity required; technology refresh cycle to be continued within the ODPS as described.

  46. Build 1 (L-18 months) All interfaces fully implemented and tested Verify initial versions of operational code ported and running in I&TSE L-3 product code developed and tested Prelaunch VIIRS test data storage and SDS interface testing support with existing ODPS storage capacity Initial test products generated for review by VIIRS Ocean Science Team Build 2 (L-12 months) Routine exercise of interfaces to acquire proxy, surrogate (Aqua?) and/or simulated data Verify pre-launch version of operational code running in I&TSE Browse and distribution capability developed and tested Test products routinely acquired as available and posted for access by VIIRS Ocean Science Team Data storage for one year Ocean PEATE Build Schedule

  47. Ocean PEATE Development Schedule

  48. External Schedule Dates • C3S –SDS Integration • October 2008 • IDPS – SDS Integration • October 2008 • NSIPS – SDS Integration • October 2008 • Functional Thread Test Dates • FTT8 A • Ingesting xDRs – October 2008 • FTT8 B • Validating xDRs – October 2008 • NPP Compatibility Tests • NCT2C - December 2007 • NCT3 - September 2008

  49. Ocean PEATE Testing • ODPS Testing Philosophy: • Test as you run. • Test early, test often. • Phased approach to testing: get one thing working and move on to the next step until entire end-to-end stream is operational. • Testing VIIRS support to be done within operational ODPS environment, with products clearly identified for separation from operational product stream. • Testing stages: • Interfaces • Internal functionality (processing, evaluation) • End-to-end • Test data sources: • VIIRS proxy data • VOST simulation • MODIS products (e.g., initial SST validation) • Availability of “official” test data is an issue.

  50. Ocean PEATERequirements Implementation • 95% of requirements (19) implemented by Build 2; add additional storage capacity before launch

More Related