The NPP Atmosphere Product Evaluation and Test Element (PEATE)

1. The NPP Atmosphere Product Evaluation and Test Element (PEATE) Liam E. Gumley, Hank Revercomb, Scott Mindock, Steve Dutcher, Robert Holz, Andy Heidinger, Mike Pavolonis, Richard Frey, Bryan Baum, Paolo Antonelli NPP Science Team Meeting Annapolis MD 22-24 August 2007

2. NPP Sensor Payload Includes: Visible Infrared Imager Radiometer Suite (VIIRS) Cross-Track Infrared Sounder (CrIS) Advanced Technology Microwave Sounder (ATMS) Ozone Mapping & Profiling Sensor (OMPS) Launch is expected in 2010 The NPP Mission for NASA: Continue the Climate Record • NPP is intended to continue the climate record established by Terra and Aqua. • NPP Science Products are known as “Environmental Data Records” or EDRs. • NPP EDR Algorithms were developed by industry, not by the NPP Science Team. Land EDRs Surface Albedo Land Surface Temperature Snow Cover and Depth Surface Type Active Fires Ice Surface Temperature Vegetation Index Aerosol Optical Thickness Aerosol Particle Size Ocean EDRs Chlorophyll Sea Surface Temperature Ozone EDRs Ozone Total Column/Profile Atmosphere EDRs Cloud Mask Suspended Matter Cloud Cover/Layers Cloud Effective Particle Size Cloud Top Height Cloud Top Pressure Cloud Top Temperature Cloud Base Height Cloud Optical Thickness Sounder EDRs Vertical Moisture Profile Vertical Temperature Profile Vertical Pressure Profile

3. What is the Atmosphere PEATE?

4. Role of the Atmosphere PEATE for NPP Q: Are the EDR algorithms and products any good? A: Need a way for the NPP Science Team to evaluate them. The Atmosphere PEATE: Assists the NPP Science Team in assessing the suitability of NPP atmosphere EDRs for continuing the NASA climate record, Provides an environment for pre-launch testing and evaluation of operational atmosphere EDR algorithms, Uses available validation data for rapid pre and post-launch assessment of NPP EDRs and SDRs, Provides an environment where the NPP Science Team can test alternative EDR algorithms on significant samples of global data. Heritage Atmosphere PEATE:University of Wisconsin-Madison MODIS/AIRS Teams Land PEATE: NASA MODIS Adaptive Processing System Ocean PEATE: NASA Ocean Data Processing System Ozone PEATE: NASA OMI Data Processing System Sounder PEATE:NASA JPL AIRS Team

5. NPP Atmosphere PEATE at SSEC Supports NPP Science Team evaluation of NPP SDR, and atmosphere EDRs Provides a testbed for improved EDR algorithms (VIIRS, CrIS, ATMS) Provides processing and visualization resources for NPP Science Team Serves as a building block for constructing in-house, long term, multi-satellite climatology of cloud retrievals (funded by multiple agencies) Facilitates the development of product centric multi-mission satellite retrieval algorithms and software Offers its resources to the atmosphere community to enable development of atmosphere and cloud CDRs

6. What is different about the Atmosphere PEATE? Old Paradigm at SSEC • Very good at processing today’s data over CONUS in real-time. • Relied on large external systems (e.g., NASA MODAPS) to process historical datasets (e.g., MODIS C5 Reprocessing). • Evaluated and validated science products in intermittent bursts. New Paradigm at SSEC (Atmosphere PEATE) • Continuous, automated, global science product evaluation. • Rapid retrospective global product generation (> 100x real-time) when required (e.g., to create CDRs, or to evaluate alternative EDR algorithms). • Automated ingest of all available validation data and comparison to satellite derived products. Key aspects of PEATE infrastructure: • Atmosphere PEATE Science Processing System • Measurement Based Product Generation Software • Satellite and Ground Based Product Evaluation

7. 1. Atmosphere PEATE Science Processing System

8. Atmosphere PEATE Science Processing System Q: Why does the Atmosphere PEATE need a science processing system? A: To enable creation of consistent long-term cloud property datasets for EDR evaluation. SDRs and EDRs obtained from IDPS will be a moving target as algorithms, LUTs, and ancillary data are debugged and refined. To evaluate climate quality of atmosphere EDRs, must use a consistent version of the calibration and science algorithms for a long-term dataset (e.g., one month, one year, entire mission). EDRs must be created rapidly in order for Science Team to give timely feedback to NPP project on algorithm performance. NPP Science Team members do not have the individual resources to host large datasets, integrate operational NPP algorithms, and test improved/alternative algorithms. The NPP cloud products must be put into context with historical and ongoing global cloud property datasets (e.g., PATMOS-X, UW-HIRS, MODIS Terra/Aqua,) to create self-consistent climate data records (CDRs).

9. Science Processing System Trade Studies and Key Decisions • We examined NASA Ocean SDPS and MODIS Land Processing Systems. • Lessons learned: • Recipe-based approach to running science algorithms (system doesn’t care what the algorithm is, as long as it knows how to assemble the ingredients to make the recipe) • Cluster of compute resources (no need for a large shared memory computer) • Decouple the components of the processing system (store, compute, distribute) • Use commodity hardware/software (e.g., Rackmount Intel/AMD servers, Linux) • Key Design Decisions: • Create a system where individual components have loose dependencies on each other. • Leverage existing cluster processing hardware infrastructure and knowledge base. • Use established software technologies where possible to speed development time (e.g, Eclipse, SOAP) • Create a system which is scalable, efficient, and cost effective.

10. Science Processing System Preliminary Design Ingest Data Manage Data Manage Processing Process Data

11. Atmosphere PEATE: Prototype Hardware (January 2007) Based on commodity hardware: Dual/Quad core servers, SATA RAID, Gigabit Ethernet. By NPP launch: 250 CPU cores, 215 TB disk. Aiming for 100x processing rate for NPP Atmosphere EDRs. 50 CPU cores 40 TB disk

12. 2. Measurement Based Product Generation Software

13. Measurement-Based Product Generation Software Old Paradigm at SSEC • Data from different sensors processed separately using separate science algorithms. • Each algorithm used different ancillary (i.e., non-satellite data) sources, interpolation, filtering etc. • Software in general was designed to work in specific compute environments (e.g., DAAC, McIDAS). • Software was not available to the community for review, modification, and enhancement. New Paradigm at SSEC • Data from different sensors processed using common science algorithms and ancillary data. • Software designed to work in generic environments (e.g., require only Linux and HDF4/5). • Software is freely available to the community (e.g., SeaDAS).

14. LEOCAT: Low Earth Orbit Cloud Algorithm Testbed LEOCAT History • Developed by Mike Pavolonis at UW under VIIRS IGS funding to investigate differences in the operational VIIRS cloud algorithms and heritage algorithms in a manner that isolates algorithmic differences. • The best way to accomplish this is to apply each algorithm to the same Level 1B and ancillary data sets using the same radiative transfer model. • A secondary use of LEOCAT is to serve as an algorithm development tool and global EDR processing system. • LEOCAT approach is also being used for GOES-R AWG work (GEOCAT). LEOCAT Features • Handles multiple imaging sensors (e.g., MODIS, VIIRS, AVHRR) • Multiple algorithms with the same and/or different parameters can be executed in one instance • Allows for the addition of new algorithms with minimal programming effort (can be added as shared libraries) • Produces HDF4 output • Can use CRTM or PLOD forward models • Can use GFS, NCEP, or ECMWF ancillary data • Optimized for efficiency • Process a single or multiple granules in one instance

15. LEOCAT Architecture LEOCAT Core VIIRS Science Algorithm

16. Cloud Mask Example: Aqua MODIS, 2006 day 240, 16:30 UTC

17. MODIS C5 Cloud Mask

18. VIIRS OPS v1.4 Cloud Mask in LEOCAT

19. 3. Satellite and Ground Based Product Evaluation

20. EDR Evaluation using Satellite and Ground Measurements • Evaluate the effectiveness of proposed cloud algorithms and the resulting global cloud products generated from MODIS (proxy for VIIRS), AIRS (proxy for CrIS), Cloudsat, and CALIPSO • A subsequent test of algorithm robustness will be to apply the cloud algorithms to METOP data (AVHRR, HIRS, and IASI) for concurrent time period as A-Train data analyses • Build the capability to assess instrument issues, such as out-of-band response, channels that perform out of spec, detector striping, etc • When VIIRS is launched, it is unlikely that a space-based lidar/radar will be in operation and there will not be continuous coincident lidar/radar measurements with VIIRS • A combined satellite and ground measurement plan provides a comprehensive evaluation capability to assess the VIIRS products

21. EDR Evaluation Measurement Plan • The NASA A-Train measurement platform using MODIS as a proxy for VIIRS will provide: • a platform to compare the VIIRS algorithms directly with MODIS, CALIPSO and CloudSat cloud retrievals (global) • a “baseline” for our global performance expectations for VIIRS • The assessment using ground measurements will provide well-calibrated point measurements that will be available at VIIRS launch • The combined ground/satellite evaluation using MODIS will provide a measure of how representative the ground evaluation will be in determining the global performance of the VIIRS retrievals at launch

22. EDR Evaluation: Cloud Height Global Images (August 2006)

23. EDR Evaluation: Total Precipitable Water • MODIS MOD07 TPW product compared to AIRS, Microwave Radiometer, and Microwave Radiometer Profiler at 9 ARM locations (6 SGP, 3 TWP): 164 matchups in August 2006. • At least 1 clear sky MOD07 pixel in a 3x3 FOV over the site MODIS TPW vs. Ground based TPW and AIRS TPW, August 2006

24. Atmosphere PEATE: Achievements to Date Processed one month of global Aqua MODIS proxy data using MODIS, MODIS VIIRS-like, and VIIRS OPS cloud mask algorithms. Collected Aqua MODIS global Level 1A data since Jan. 2006. Adapted existing software (LEOCAT) to create infrastructure for running VIIRS OPS EDR code on Linux (32, 64-bit) and OS X. Demonstrated 50x processing rate on first generation computing system. Created validation plan and demonstrated a validation approach for cloud mask using CALIPSO CALIOP. Completed System Requirements Review for NASA. Demonstrated process for evaluating a VIIRS EDR (Cloud Mask). Completed successful Preliminary Design Review for NASA. Started deployment and testing of Science Processing System.

25. Atmosphere PEATE: Summary • Atmosphere PEATE has designed an infrastructure, methodology, and science processing system to support the NPP Science Team in evaluating Atmosphere EDRs. • Atmosphere PEATE Science Processing System will be available to generate consistent long-term EDR datasets as needed to evaluate the climate quality of NPP Atmosphere EDRs, and to compare with historical CDRs including those from MODIS, AVHRR, and HIRS. • Provides an opportunity to extend measurement-based approach for generating cloud products to other satellite sensors datasets including GEO. • SSEC is seeking additional funding to leverage NPP Atmosphere PEATE for producing other Cloud Climate Data Records. • Liam.Gumley@ssec.wisc.edu