Scatterometer Algorithm and Software Status Update: March 2010 (APF)

1. Adam Freedman Simon Yueh Alex Fore M. Julian Chaubell Jet Propulsion Laboratory Aquarius Algorithm WorkshopMarch 9-11, 2010Santa Rosa, CA Scatterometer Algorithm and Software StatusUpdate: March 2010 (APF)

2. Scatterometer Data ProcessorL1A_to_L2

3. Key Scatterometer Data Processing Requirements • Locate each Sigma0 on Earth. • Convert the L1A Aquarius data from counts to calibrated normalized radar cross-sections (Sigma0) • Generate an error estimate (Kpc) for Sigma0. • Incorporate quality control flags (RFI, land fraction, etc) • Generate Tb corrections and wind speed estimates for surface roughness effects (L2)

4. Documentation • Scatterometer Science Processing Software (L1A_to_L2) User Manual • Project Document: AQ-485-0541; JPL Document: D-51444 • Draft v.1.4, 02 March 2010 • Preliminary or Initial Release after Algorithm Workshop • Simulator Design • aq_scat_ant_memo: processing of antenna patterns • Aquarius_Software memo • PowerErrors memo • Aquarius Downlink Block Simulator memo • Simulator Algorithms • SmThPhases memo – beam pattern phase discussion • 334-JCH-20071204-014, Inversion algorithm and K factor memo • 334-JCH-20071129-013, Scalar Radar Equation and Polarimetric Radar Equation memo • ScatSimPhysics.pdf memo • Inversion Design • Aquarius Radar Processing v_07 (old) (algorithm document: needs updating) • L2_processing.doc memo

5. L1A to L1B Algorithm Structure Read ephemeris/attitudes. Compute cubic spline coeffs. L1A Input file Input flags Info/Debug files Read radar science data. Read instrument telemetry. Quality check of data. Compute look vectors. Compute LB-corrected echo power into antenna. Compute SNR, Kpc. Use K-factors to compute geolocated sigma-0. Compute land, ice fractions. Compute flag info. Write L1B Output. K-factor tables.Land mask. To L2 processing Antenna patterns (6) Sea ice data HDF-4 Level 1B File (includes processing flags) Instrument parameters

6. L1A to L1B Algorithm Status

7. Contents of Level 1B HDF-4 Output File There is one set of values for each 1.44-second block of data returned by the Aquarius Instrument. For a single orbit, there are typically about 4080 data blocks; there are thus about this number in each single-orbit HDF-4 output file.

8. Level 1B Quality Flags (a)

9. Level 1B Quality Flags (b)

10. L1B to L2 Algorithm Structure From L1B processing L1B Input file Input flags RFI filtering of data. Time averaging of data. Top of ionosphere NRCS. Correct for Faraday rotation. Correct for cross-pol leakage. Top of atmosphere NRCS. Compute total power. Propagate to surface. Correct for land or ice in beam. Use model wind directions (and wave slope/height TBD) plus sigma-0 to estimate wind speeds and delta-Tb. Propagate uncertainties. Info/Debug files Ocean winds.Ocean waves (NOAA WaveWatch 3). Ionosphere.Magnetic field. L2 Output Text Files Beam data. Wind model coefficients. HDF-5 Level 2 File Faraday Rotation from Radiometer

11. L1B to L2 Algorithm Status * Indicates capability not fully tested or not yet delivered to GSFC

12. Parameters Generated By Level 2 Code

13. . Scatterometer Data for Aquarius Level-2 Products

14. Level 2 Quality Flags scat_rfi_flags (byte, array size Number of Blocks x Number of Beams x Number of RX polarizations): long_name = “Scatterometer RFI flags”; Radio frequency interference flags for each scatterometer measurement in the block. There is one flag per sequence per receive polarization, in the sequence order (MSB) sequ. 8, sequ. 7, …, sequ. 1 (LSB), and polarization order HV, VV, VH, HH. (L2_scat_RFI_flags) scatterometer_flags (4-byte integer, array size Number of Blocks x Number of Beams): long_name = "Scatterometer data quality flags"; each bit or subset of bits represents a data quality condition that was detected for that beam and block. Some flags are valid for all channels, while others have bits corresponding to each of the 4 polarization channels (in order HV, VV, VH, HH). Some flags are two bit quality indicators, with the first bit indicating moderate contamination and the second bit, severe contamination, as described below. (L2_flags)

15. L1a_to_L2.f Processor Outline (delivered version)

16. L1a_to_L2 Operating Flags: Required • “-i <filename_string>”: Tells the software to process the level-1A data found in the HDF5 file “filename_string”. • “-param_file <filename_string>”: Tells the software to load the system parameters from the file “filename_string”. It is a good idea to always use the full path to the file as opposed to a relative path. • “-limits <filename_string>”: Tells the software to load the L1B limits table from the file “filename_string”. It is a good idea to always use the full path to the file as opposed to a relative path. • “-dir_dat <Aquarius_data_directory>”: Shall be set to the ‘Aquarius_data’ directory that contains ancillary data used for the forward simulation and L1A_to_L2 processing. • “-dir_out <output_directory>”: Shall be set to the pre-existing directory where the user wants that L1B and L2 output files be written. • “-dir_scratch <scratch_directory>”: Shall be set to a pre-existing directory where the software can write temporary files. If you run more than one instance of “L1_to_L2” each process should have its own scratch directory.

17. L1a_to_L2 Operating Flags: Optional “-debug <integer>”: Throughout the software there are debugging statements that write to standard output. Typically they write at debugs levels 1 and 2. Nominal usage is a debug level 0. Error messages and warnings will always be printed. -“sm_patts”: The software will use the measured scale-model antenna patterns. If not specified the software will use the simulated antenna patterns. “-l1b”: Turns on the level-1B (HDF4) output. Default is to not write L1B output. “-cal_level <1,2,3>”: Tells the software which calibration level to use. Cal level 1 does not apply any temperature-dependent loss correction. Cal level 2 applies a temperature-telemetry-dependent loss correction for those RF zones outside the loopback calibration. Cal level 3 applies temperature-dependent corrections to all RF components. “-read_sci_prefix <read_sci_prefix_string>”: Tells the software to use an alternative input stream for our L0 reader. Requires that “-emph_XYZ” and “-emph_RPY” also be commanded. “-ephem_XYZ <XYZ_ ephemeris _file>”: Specify ephemeris file for use; only to be used with the “-read_sci_prefix” option. “-ephem_RPY < RPY _ ephemeris _file>”: Specify attitude file for use; only to be used with the “-read_sci_prefix” option. “-write_telem <telemetry_out_directory>”: Tells the software to write out the converted telemetry to the directory “telemetry_out_directory”. “-suppress_tlm_warnings”: Tells the software to suppress out-of-range telemetry warnings. Useful for processing exaggerated temperature data; without it the software will generate ~100 of megabytes of out-of-range warnings per orbit. “-L2_allow_bad”: Tells the software to allow data that has been flagged as bad at the L1B level into the block averaged L2 Sigma0. The default is to not allow data that has been flagged as bad into the L2 Sigma0 averages. “-L2_filter_rfi”: Tells the software to NOT allow data flagged as contaminated with RFI at the L1B level into the L2 averaged sigma0 computation. The default is to allow RFI data in the averages.

18. L1a_to_L2 Operating Files • Input files • The only input files the software needs is the L1A HDF5 file, the L1B_limits file, and the parameters file. For our own use we have an alternative input stream which uses the output of ‘read_sci’ (which reads a Level 0 or raw downlink file) and the ephemeris instead of the L1A HDF5 file. Both the L1B_limits file and the parameters file are included in the source distribution. • Ancillary Data • We will supply a copy of the ‘Aquarius_data’ directory which contains: • Geophysical model function table. • Scale-model and theory antenna patterns. • Kfactor tables for both scale-model and theory antenna patterns. • Sea ice data product and shell script to extract data from the GRIB file and write it to a temporary file that the software loads in. • Land mask. • Output files • The software will create output files in the directory specified with the “-dir_out” command line argument. The L1B output is in HDF4, and will only be written if the command line argument “-l1b” is given. The L2 output is currently written to text files and will always be written.

19. Development • Complete and deliver V1 L1A-L1B processor by the end of September ‘09 • Complete coding by the end of August • Complete testing in September using simulated L1A data • Complete coding and testing of L2 scatterometer processor by the end of September using simulated JPL L1B data in HDF4 format • Complete coding of L2 scatterometer processor using simulated Aquarius L1B data (HDF5) by the end of October ‘09. • Delivery of L1A_to_L2, Version 1, November ‘09. • Delivery of Version 2 of L1A_to_L2 by April/May ‘10. • TOI -> TOA correction (for polarimetric products) • Uncertainty terms, TOA products, L2 flags, etc. • Perturbations and Mission-Ops exercises, Summer ’10. • Testing data flags and data checking algorithms • Need complete set of ancillary data

20. L2 Issues/Questions • Synchronizing radiometer and scatterometer data using? • Block ID • Time tag • Location beam center • Transfer of Faraday Rotation information between rad and scat • Development and testing of off-nominal processing capabilities • Part of pre-launch testing • Errors in position, pointing, cold-sky operation • Who generates simulated data sets? When? • Model improvements • Before launch: Using simulated data • After launch: procedures and timelines, using actual data • L2 data flagging • How should flagged data be processed or represented? • Rain flag source? • Radiometer/Scatterometer consistency in L2 products?

21. Scatterometer Simulator

22. Scatterometer Simulator Structure Orbit ephemeris & attitude (MJ2K,ECF) Read position/attitude data Compute spline coeffs Red items in development Input flags K-factor tablesLand mask Compute land backscatter level Land cover type Soil type Soil moisture Soil temp ASCII output data Compute sigma-0 Compute Faraday Rotation Integrate over beam w/ land and ice included Compute Xint Write (one-time) K-factor and land mask tables Antenna patterns (6) Ocean windsIonosphereMagnetic fieldLand maskSea ice data Science Block Writer Add Gaussian noise Interpolate to PRIs Convert to DNs Wind-backscatter model function Compute instrument state Compute Xcal Compute measured echo and noise powers Compute other block data Binary block data Instrument parameters & temperatures GSFC/L0/L1A simulation

23. Forward Data Simulation: One Orbit

24. Input Powers to L0 Simulator With No Faraday Rotation With Faraday Rotation

25. Data Read From L0 Block Data File

26. Simulator Status

27. Simulator Status (continued)

28. Combined Data Intercomparisons

29. Scat vs Rad L1A Comparison Who will do these comparisons and checks? When? L1B, L2?

30. Sigma-0: L1B, L2 TOI, Total Power

31. Land & Ice Fractions

32. Scatterometer RFI Detection

33. Signal histogram Noise floor trigger count threshold RFI signal Scatterometer RFI Detection Scheme • Challenge: To pick a scheme and set of parameters that will be as sensitive as possible to short-duration, sinusoidal RFI signals without being triggered by Gaussian noise floor of receiver • Use ADC detector sampling of receive power at 16 MHz to set an RFI flag; samples that exceed specified power and count level should be flagged as probable RFI • Effectively a coarse PDF computation, looking for small number of excess outliers to flag RFI • Operates only in “Noise-only” pulses • Echo from ocean will not trigger flag, thresholds can be fixed, independent of ocean return • Noise-only windows interspersed with echoes, no more than 15 msec from echo window • Adjustable threshold for sensitivity and trigger count for time resolution, set via ground commands • (Radiometer correlated noise diode (CND) firing may frequently trigger RFI flag)

34. Example: Measured Power and RFI Flag Noise floor (-34.4 dBm) Noise + CW five dB less Noise + CW three dB less Noise + CW one dB less Noise + CND Measured Echo RFI Flag Injected CW 5 dB below noise floor does not trip flag, 3 dB below floor intermittent tripping, 1 dB below floor continuous tripping CND noise-like signal, 3.5 dB below noise floor, triggers RFI flag most of time

35. Example: Six Channels for RFI Detection • Flight software patch allows RFI to be monitored separately for V and H channels for each of three beams, reducing effect of CND-induced false triggering of RFI flag. • RFI of different characteristics can be mapped by varying threshold parameters

36. Uncertainty Estimation

37. Kpc Model and Example • Kpc computed for individual measurements based on SNR, radar parameters, and geometry • Kpc for block-averaged data computed from variances, with “good-measurement” mask • Kpc for total power computed from sum of variances Individual measurements Block measurements Total power measurements

38. Kpc: L1B, L2, Total Power

39. Brightness Temperature & Wind Speed

40. Plans for Commissioning, PLAR, and L+6 Months

41. Plans for Commissioning and PLAR • Commissioning objectives • Verify that instrument hardware is operating as expected • Especially temperature stability • Verify that operating temperature region is “glitch-free” • Verify that no “obvious” inter-instrument interference • Limited “performance” objectives • PLAR requirements • Reflector deployed in expected configuration • Gross pointing offset • Temperature stability achieved

42. Plans for Subsequent ~6 Months • Performance validation • Primarily 7-day stability • May not address annual variability • Calibration site verifications • Calibration offsets and drifts • Model function improvement • Improvement of wind retrieval algorithm and delta TB error estimates • Data quality flagging

43. Open Issues

44. Open Issues • Antenna Pattern cross-pol leakage for accurate recovery of cross-pol terms • Does not affect primary algorithm using total power products • Science analysis tools • Time series analysis for ground calibration locations • Crossover analysis • Land/ocean crossing & pointing verification • Geographic plotting and ancillary data-set overlays

45. BACKUP

46. Run Times • Machine: “caret” • 1.0 GHz, 4 cpu cores • Simulator: ~1.4 sec/orbit position with Faraday rotation on • About 24 cpu-hours for one day of simulated data • L1A_to_L2: 10 minutes per orbit; ~one month to process one year • Machine: Mac Pro • 2.8 GHz, 8 cpu cores • Simulator: ~0.8 sec/orbit position with Faraday rotation on • About 13 cpu-hours (one cpu) for one day of simulated data • Parallel processing using Mac Pro 8 cores • Simulator: About one week of data every night • One to two calendar months to produce one year of simulated data, depending on machine availability • TBC Option using Saltine for parallel processing • TBC new computer: Aquascat (8 processors, 3 GHz) • Use during Commissioning, and afterwards for 6-months and scat science.

47. Temperature Variation Input Table

48. Scatterometer Parameter Table

49. HKT Telemetry for L1B File

50. Simulator Block Diagram Input simulator command options Orbit ephemeris file S/P XYZ position vs time, either ECF or MJ2K Read orbit and attitude files Compute cubic spline coefficients Antenna beam pattern files Complex Eco and Ecr, (theta,phi,beam#,H/V-pol) S/P Attitude file S/P RPY vs time, or equivalent Read antenna gain, compute arrays of two-way relative and max gain Next orbit position Compute x, v, a, at t in ECF for one epoch Compute S/P nadir location Compute S/P local frame orientation Scatterometer HW data file Tx power, gains, losses, temperatures Instrument Simulator Compute Xcal_sim, LB power, noise power, DC offset Integration Over Beam Pattern Compute Xint, beam-weighted sig-0, Mean sig-0 over 3-dB footprint Next beam Compute footprint location, Incidence, azimuth angles Compute simulated measurement HV,nV,VV,VH,nH,HH,LB,DC vs t Write to output files Time, Echo, Noise-only, LB data per beam, telemetry