Topic: Discrepancy Reports Generation Coordinated Efforts to Expedite Algorithm Change Process

1. On-Going STAR ActivitiesIn Collaboration with Cal/Val Team Members Topic: Discrepancy Reports Generation Coordinated Efforts to Expedite Algorithm Change Process Degui Gu, Xia L Ma, and Denise Hagan Northrop Grumman Aerospace Systems Xu Liu and Susan Kizer Langley Research Center Murty Divakarla, Mike Wilson, Xiaozhen Xiong, Changyi Tan, Nick Nalli, Eric Maddy, and Antonia Gambacorta STAR, NOAA/NESDIS Wael Ibrahim Raytheon Other Cal/Val Teams Other Teams (Richard Cember ) As Directed by: Chris Barnet, CrIMSS Cal/Val Team Lead

2. Coordinated EffortsNGAS, LaRC, STAR, and Other Cal/Val Teams • A thorough understanding of the CrIMSS EDR algorithm and the code to emulate/test various flavors of IDPS CrIMSS EDR algorithm • MX 5.3, MX 6.3 (October 10th), MX 6.4 (January 2012) . • CrIMSS EDR off-line Algorithm/G-ADA • ADL-Based Instances (AIT TEAM) and coordinated Testing • Generate and utilize a variety of validation data sets • Validate CrIMSS EDRs • Verify LUTs currently in use and provide updates and Improvements • Generate Discrepancy Reports (DRs), Attend to DRs generated by CrIMSS cal/val team members, verify the validity of the DRs to expedite Algorithm Change Process • We consider this Very Important • Supporting Cal/Val TEAMs • Demonstrate Provisional Maturity of the CrIMSS EDR Algorithm

3. Objectives of this Presentation • >> A Short discussion on the DR Process. • Ensure that DR submitted has a positive impact • (a) Yield (b) Accuracy/Precision (c) Cross-Algorithm Effect • (a) Mw-only (b) IR+MW (c) Everywhere • (a) Low (b) Moderate (c) Major • Solicit any DRs on the CrIMSS EDR Performance • Suggest a work plan on the DRs • IDPS Builds Info • Mx6.3 Transition To OPS (TTO) 10/8 or 10/12 • Mx6.4 TTO 2/18/12 (Per Alex Delfierro's request - RTN didn't agree yet) • Mx7 Maintenance Readiness Review (MRR) 3/21/13 • DR submission for the MX6.4 Build • October 20 -> To STAR CrIMSS algorithm group • November 20, to DPA/DPE • Example (Mike Wilson’s Presentation)

4. Algorithm Change ProcessExcerpt from: Integration and Transition to Operations(Presented by Kerry Grant, Raytheon, September 19, JPSS TIM Meeting) Discrepancy Report Assessment Team (DRAT) Algorithm Engineering Review Board (AERB) Operations and Support (O&S), Mission Operations Team (MOT), Operational (Ops)

5. Algorithm Change ProcessExcerpt from: Integration and Transition to Operations(Presented by Kerry Grant, Raytheon, September 19, JPSS TIM Meeting)

6. Outline of Investigations for DR Process • Background • Rationale for the proposed changes • Test Plan • (a) CrIMSS-Offline EDR Code (b) GRAVITE-ADA (c) ADL • Maturity (a) Well Tested (b) In the Process (c) Beginning phase • Required support for the cal/val group members • STAR can help you direct to the right personnel. • Data needs • Type of Change • (a) Fast Track (LUTs) (b) Long Track (Algorithm Code changes) • Size of change • (a) Minor (b) Moderate (c) Major • Impact on CrIMSS EDRs • (a) Yield (b) Accuracy/Precision (c) Cross-Algorithm Effect • (a) Mw-only (b) IR+MW (c) Everywhere • (a) Low (b) Moderate (c) Major • EXAMPLE (Mike Wilson’s Presentation)

7. Example DR Evaluation: Surface Pressure • Mike Wilson, Murty Divakarla, Xiaozhen Xiong, Changyi Tan, I. M. Systems Group • C.D. Barnet, STAR NOAA/NESDIS • Sung-Yung Lee, JPL • (Priority – 1) • LT_A. Description:Surface pressure values were spiking above 1100 mb at certain locations in the IDPS product (SYL) • Background and rationale: Surface pressure is a derived from the GFS forecast/analysis fields interpolated in time and space to the locations of the CrIS FOVs. This ancillary data is used as one of the boundary conditions to run the CrIMSS EDR algorithm. • Test Plan (off-line/G-ADA/ADL): Combination of Offline and IDPS EDRs, Focus Day (05/15) • Results: Evaluated using IDPS EDR products, and the Off-line EDR products with surface pressure ancillary data generated through two different approaches. • Maturity/Consensus among cal/val members: Okay, Tested; Yes • Type of change /Size of Change/Code affected: Long Track, potentially major, since we can’t see it, Yes • Impact (Marginal/Moderate/Major): MW+IR algorithms both, marginal. (Affects which levels get reported, slightly impacts temperature and water vapor products). May have impact on other retrieval algorithms (?) OMPS (?)

8. Example: Surface Pressure • Background: • Sung-Yung Lee discovered that surface pressures were spiking above 1100 mb at certain locations (e.g. northern end of South America). This was in the IDPS product. • Test Plan: Combination of Offline and IDPS • Need to determine: • Was this a problem in the off-line code as well? • Was it limited to a few points above 1100 mb, or more widespread? • How much did it impact the product? • Was this an easy fix (LUT), code fix, or beyond the EDR?

9. Versions of Offline EDRAvailable to Us: • MX6_Baseline: • Offline EDR being used to mimic the current IDPS product (MX5.3). MW bias turned off, all other files are used in IDPS. • Surface pressure and land fraction forced to match IDPS. • (This modification was done because of this investigation.) • Intent is to show that we can emulate the IDPS, so we can say the improvements to the offline mirror the expected improvements in the operational algorithm. • MX6_Baseline + CLIM/BIAS LUTs from Degui Gu. • Used to mimic the expected future IDPS release on October 10th. • MX6_Baseline + All of Xu Liu’s Modifications. • Used to test possible future updates suggested by Xu Liu. • Used to compare Xu Liu’s updates to NGAS LUT updates. • ADL: Used to mimic the operational environment.

10. Beginning Phase Testing • Is this a problem with offline? • For 2318 Granules: • About 20 profiles with IDPS > 1050 mb. • No profiles from Offline > 1050 mb. • Tended to find extremes in IDPS more frequently (both very low and very high surface pressures)

11. Beginning Phase Testing IDPS Ocean IDPS Land 0.15 (0.943) • Problem over Land and Coast, not Ocean. 0.10 Probability Density 0.05 0.00 -5 0 5 -20 0 20 IDPS surface pressure minus Offline surface pressure.

12. Beginning Phase Testing • (From Xiaozhen Xiong) • Largest differences over coast/mountainous areas.

13. Testing: “In The Process.” • What does this impact? By how much? • Tested 104 Granules offline, using: • IDPS • Offline using our land fraction and surface pressure (done by Xiaozhen). • Offline pulling land fractions and surface pressure from the IDPS EDR product.

14. Mean temperature lines on top of each other. • Temperature biases within 0.1 K. • Similar impact on water vapor. • Main impact of bug is that it impacts which levels get reported.

15. How to fix bug: • Surface pressure is interpolated to the CrIS FOV outside of the EDR. • Land fraction is read from an external file which does not match our local version. • We know where these variables enter the EDR, but the fix is external to the EDR.

16. Sample Checklist: • Background: Bug discovered by Sung-Yung Lee. • Rationale for proposing change: Surface pressures are not naturally above 1100 mb. • Test plan: IDPS vs. Offline done. Need access to NWP to fully address. GRAVITE? • Maturity: In the process. Shown the extent of the problem, but we don’t have the solution to suggest in the DR, which means it will take more effort on others’ parts to fix. • Type of change: code change outside of EDR. • Size of change: potentially major, since we can’t see it. • Impact: MW+IR algorithms both, minor impact. (Affects which levels get reported, slightly impacts temperature and water vapor products)

17. Potential DRs –LUT Updates (Priority – 1) • Candidate LUT updates • LT_A. CrIS RTM error LUT update (priority: 1) • LT_B. CrIS sensor error LUT update (priority: 1) • LT_C. ATMS SDR bias correction LUT update (SDR antenna efficiency correction; water vapor channel improvements) (priority 2) • LT_D. CrIS RTM bias correction LUT update (SW band improvements) (priority 2) • LT_E. ATMS RTM error LUT update (priority 3) • LT_F. ATMS sensor error LUT update (priority 3) • LT_G. ATMS remap SDR noise amplification factor LUT update (priority 3) • LT_A. Description:Updating CrIS RTM error LUT which contains IR OSS RTM error estimates (NGAS) • Background and rationale: Current LUT is populated with pre-launch conservative estimates. Analysis of on-orbit data indicated the OSS RTM is more accurate and as a result, this LUT needs to be updated to fully exploit CrIS data and improve CrIMSS EDR product quality • Test Plan (off-line/G-ADA/ADL): NGAS-offline science code, Focus Day 05/15/ Data • Results: Tested using NGAS offline science code using GD 05/15/12 dataset to demonstrate positive impact. Previously tested on G-ADA using MX5.3; Need to re-test with the latest baseline code (MX6.2) on new ADA environment • Maturity/Consensus among cal/val members: Okay, Tested; None • Type of change /Size of Change/Code affected: Fast Track, Moderate, None • Impact (Marginal/Moderate/Major): Positive

18. LT_B.P1 Description:Updating CrIS sensor error LUT which contains estimated errors in CrIS SDR radiances (NGAS) (Priority – 1) • Background and rationale: Current LUT is populated with pre-launch conservative estimates based on CrIS sensor requirement specification. Analysis of the on-orbit data has shown CrIS sensor performance is better than spec. This LUT needs to be updated to fully exploit CrIS data and improve CrIMSS EDR product quality. • Test Plan (off-line/G-ADA/ADL): NGAS-offline science code, Focus Day 05/15/ Data • Results: Tested using NGAS offline science code using GD 05/15/12 dataset to demonstrate positive impact. Previously tested on G-ADA using MX5.3; Need to re-test with the latest baseline code (MX6.2) on new ADA environment • Maturity/Consensus among cal/val members: Well Tested, Yes. • Type of change /Size of Change/Code affected: Fast Track , Moderate, None • Impact (Marginal/Moderate/Major): Positive.

19. LT_C.P2 Description:Updating ATMS SDR bias correction LUT which contains coefficients for removal of ATMS scan-dependent biases (NGAS, LaRC, STAR) (P-2) • Background and rationale: Current LUT (MX6.3) was derived based on ATMS TDRs which didn’t have antenna efficiency correction. Also additional testing using Golden Day 5/15/12 data seems to indicate residual biases in water vapor sounding channels (18-21). Need to be updated to be in synch with ATMS SDR team update to their PCT update for scan dependent bias correction. • Test Plan (off-line/G-ADA/ADL): Yet to Evolve • Results: Not updated LUT yet, awaiting ATMS SDR input. Previously tested on G-ADA using MX5.3; Need to re-test with the latest baseline code (MX6.2) on new ADA environment • Maturity/Consensus among cal/val members: Not Tested, Fast Track • Type of change/Size of Change/Code affected: Moderate, None • Impact (Marginal/Moderate/Major): • LT_D.P2 Description:Updating CrIS RTM bias correction coefficient LUT which contains coefficients for correction of CrIS IR forward model bias errors (NGAS, LaRC, STAR) (P-2) • Background and rationale: Current LUT (MX6.3) was derived based on LaRC estimates. LW and SW appear to have a little discrepancy. The update may be desired, but its impact on CrIMSS EDR performance is expected to be minimal • Test Plan (off-line/G-ADA/ADL): • Results: • Maturity/Consensus among cal/val members: Additional tuning is available based on NGAS analysis of GD 05/15/12 dataset • Maturity/Type of change: Not Tested, Fast Track, • Size of Change/Code affected: Moderate, None • Impact (Marginal/Moderate/Major): Marginal

20. LT_E-G P3 Description:Updating ATMS RTM error LUT, ATMS sensor error LUT, and ATMS remap SDR noise amplification factor LUT which combined to characterize the uncertainty of ATMS SDR radiances (NGAS) (P- 3) • Background and rationale: Current LUTs (MX6.3) were based on pre-launch estimates which appear to be a little conservative. A tuning to these LUTs can be done to more accurately characterize the error in ATMS radiances to potentially improve CrIMSS EDR performance • Test Plan (off-line/G-ADA/ADL): • Results: Analysis underway, and preliminary results seem to indicate the combined effects of the current LUTs are reasonable • Maturity/Consensus among cal/val members): Not Tested • Type of change/Size of Change/Code affected: Fast Track, Moderate, None • Impact (Marginal/Moderate/Major): ???

21. Potential DRs –Candidate code updates • CD_A. Index for non-LTE is off by 22 ATMS channels (priority: 1) NGAS • CD_B. Handling of sensor and RTM noise errors (priority: 1) NGAS • CD_C. Logic for QC (priority 1) NGAS • CD_D. Tskin – Tair constraint change in CLIM LUT (LaRC) (priority 1) • CD_E. Logic for ocean prior selection (priority 1) (LaRC) • CD_A.P1. Description:Indexing error of N-LTE affected channels (index didn’t account for ATMS channels) (NGAS) • Background and rationale: Current ops code has an error in specifying the channels affected by N-LTE that should not be used in the retrieval. The coding error needs to be corrected to prevent negative impact on CrIMSS EDR performance during daytime • Test Plan (off-line/G-ADA/ADL): NGAS-offline, ADA • Results: Code change identified and tested using both NGAS offline science code and ADA operational code • Maturity/Consensus among cal/val members): • Type of change/Size of Change/Code affected: Fast Track, Small, Yes. • Impact (Marginal/Moderate/Major):

22. CD_B.P-1Description:Atmospheric noise and sensor noise need to be combined in inverting state parameters and evaluating the radiance residuals (NGAS) (Priority – 1) • Background and rationale: Current ops code has errors in computing the total error in radiances (either from RTM forward simulation or sensor error). The coding errors need to be corrected to achieve optimal CrIMSS EDR performance • Test Plan (off-line/G-ADA/ADL): NGAS-offline science code. • Results: Code changes identified and tested using NGAS offline science code • Maturity/Consensus among cal/val members):, • Type of change/Size of Change/Code affected: Fast Track, Moderate, None • Impact (Marginal/Moderate/Major): • CD_C.P1 Description:QC not optimized to produce best CrIMSS EDR data products (NGAS) • Background and rationale: Ops code produces EDRs from MW only or IR+MW combined retrievals based on a number of chi-square tests. If a retrieval doesn’t pass the tests, the IR+MW combined retrieval is ignored and the MW only retrieval is used to generate the EDR product. However, examination of EDR quality vs. chi-square values has shown that based on the current chi-square threshold values, a good number of quality IR+MW retrievals are thrown away. Need to refine the logic and tune the threshold values to produce best EDR data products • Test Plan (off-line/G-ADA/ADL): NGAS- offline science code • Results: Preliminary code changes identified and tested using NGAS offline science code • Maturity/Consensus among cal/val members): Not Tested, Fast Track, • Type of change/Size of Change/Code affected: Fast Track, Small, Yes • Impact (Marginal/Moderate/Major):

23. CD_D.P-1Description:Tsurf-Tair constraint change in CLIM LUT (LaRC) (Priority – 1) • Background and rationale: During the day over land, the temperature differences between Tskin and Tair are pretty large, the current CLIM LUT has a very tight constraint with regards to these two parameters which leads to the warm bias of the retrieved air temperature near surface. We have relaxed the constraint and a priori based on the analysis of the ECMWF field and obtained better EDR performances. • Test Plan (off-line/G-ADA/ADL): NGAS-offline science code, ADL, G-ADA • Results: Preliminary code changes implemented in the ported IDPS code and improvement tested using both sample and global data. • Maturity/Consensus among cal/val members): Validate among cal/val team. • Type of change/Size of Change/Code affected: Fast Track, Small, Yes • Impact (Marginal/Moderate/Major): Moderate • CD_D.P1Description: Logic for Ocean prior, warm ocean test (Xu) (LaRC) (Priority – 1) • Background and rationale: for ocean climatology covariance selection, the current IDPS code select the warm ocean covariance based on either retrieved MW tskin is greater than 290 K or delta emissivity (between channel 22 and channel 1) is greater than 0.1. This logic will lead to the selection of warm covariance matrix even when we are close to polar region (there are some ice emissivity with positive delta Emissivity listed above). • Test Plan (off-line/G-ADA/ADL): offline science code. Currently, we are not using stratified CLIM covariance for MW retrieval. Need to modify the logic and threshold to better classify the surface and atmosphere clim cov. • Results: In the process of verification • Maturity/Consensus among cal/val members): Not Tested • Type of change/Size of Change/Code affected: Fast Track, Small, Yes • Impact (Marginal/Moderate/Major): moderate

24. ADL and IDPS BuildsExcerpt from: Integration and Transition to Operations(Presented by Bryan Henderson, Raytheon, September 19, JPSS TIM Meeting) ADL 4.1 ------> Equals IDPS MX6.3 Build

25. Integration and Transition to Ops(Excerpt from Kerry Grant’s talk, Sept. 19th)

26. How Do We Move forward • “what and how" of proposed changes can be expedited through Algorithm Change Process • Make sure the DR submitted is worthy of consideration • Scientific evaluation of the DR. • Provide preemptive answers to the queries expected in the implementation process

27. Backup slides