Quality, Uncertainty and Bias – by way of example(s)

1. Quality, Uncertainty and Bias – by way of example(s) Peter Fox Data Science – ITEC/CSCI/ERTH-6961 Week 11, November 13, 2012

2. Where are we in respect to the data challenge? “The user cannot find the data; If he can find it, cannot access it; If he can access it, ; he doesn't know how good they are; if he finds them good, he can not merge them with other data” The Users View of IT, NAS 1989

3. Definitions – atmospheric science • Quality • Is in the eyes of the beholder – worst case scenario… or a good challenge • Uncertainty • has aspects of accuracy (how accurately the real world situation is assessed, it also includes bias) and precision (down to how many digits)

4. Definitions – atmospheric science • Bias has two aspects: • Systematic error resulting in the distortion of measurement data caused by prejudice or faulty measurement technique • A vested interest, or strongly held paradigm or condition that may skew the results of sampling, measuring, or reporting the findings of a quality assessment: • Psychological: for example, when data providers audit their own data, they usually have a bias to overstate its quality. • Sampling: Sampling procedures that result in a sample that is not truly representative of the population sampled.

5. Data quality needs: fitness for purpose • Measuring Climate Change: • Model validation: gridded contiguous data with uncertainties • Long-term time series: bias assessment is the must , especially sensor degradation, orbit and spatial sampling change • Studying phenomena using multi-sensor data: • Cross-sensor bias is needed • Realizing Societal Benefits through Applications: • Near-Real Time for transport/event monitoring - in some cases, coverage and timeliness might be more important that accuracy • Pollution monitoring (e.g., air quality exceedance levels) – accuracy • Educational (users generally not well-versed in the intricacies of quality; just taking all the data as usable can impair educational lessons) – only the best products

6. Producers Consumers Quality Control Quality Assessment Fitness for Purpose Fitness for Use Trustor Trustee

7. Quality Control vs. Quality Assessment Quality Control (QC) flags in the data (assigned by the algorithm) reflect “happiness” of the retrieval algorithm, e.g., all the necessary channels indeed had data, not too many clouds, the algorithm has converged to a solution, etc. Quality assessment is done by analyzing the data “after the fact” through validation, intercomparison with other measurements, self-consistency, etc. It is presented as bias and uncertainty. It is rather inconsistent and can be found in papers, validation reports all over the place.

8. 20080602 Fox VSTO et al.

9. Level 2 data

10. Level 2 data • Swathfor MISR, orbit 192 (2001)

11. Factors contributing to uncertainty and bias in L2 • Physical: instrument, retrieval algorithm, aerosol spatial and temporal variability… • Input: ancillary data used by the retrieval algorithm • Classification: erroneous flagging of the data • Simulation: the geophysical model used for the retrieval • Sampling: the averaging within the retrieval footprint

12. Level 3 data

13. What is Level 3 accuracy? It is not often defined in Earth Science…. • If Level 2 errors are known, the corresponding Level 3 error can be computed, in principle, but… • Processing from L2L3 daily  L3 monthly may reduce random noise but can also exacerbate systematic bias and introduce additional sampling bias • Quality is usually presented in the form of standard deviations (i.e., variability within a grid box), and sometimes pixel counts and quality histograms are provided. QC Flags are rare (MODIS Land Surface). • Convolution of natural variability with sensor/retrieval uncertainty and bias – need to understand their relative contribution to differences between data • This does not solve sampling bias

14. Same parameter Same space & time MODIS vs. MERIS MODIS MERIS Different results – why? A threshold used in MERIS processing effectively excludes high aerosol values. Note: MERIS was designed primarily as an ocean-color instrument, so aerosols are “obstacles” not signal.

15. Why is it so difficult? • Quality is perceived differently by data providers and data recipients. • There are many different qualitative and quantitative aspects of quality. • Methodologies for dealing with data qualities are just emerging • Almost nothing exists for remote sensing data quality

16. ISO Model for Data Quality

17. But beware the limited nature of the ISO Model Land Surface Temperature anomaly from Advanced Very High Resolution Radiometer trend artifact from orbital drift discontinuity artifact from change in satellites Q: Examples of “Temporal Consistency” quality issues? ISO: Temporal Consistency=“correctness of the order of events”

18. Going Beyond ISO for Data Quality • Drilling Down on Completeness • Expanding Consistency • Examining Representativeness • Generalizing Accuracy

19. Drill Down on Completeness • Spatial Completeness: coverage of daily product Due to a wider swath, MODIS AOD covers more area than MISR. The seasonal and zonal patterns are rather similar

20. MODIS Aqua AOD Average Daily Spatial Coverage By Region and Season This table and chart is Quality Evidence for the Spatial Completeness (Quality Property) of MODIS Aqua Dataset

21. Expanding Consistency • Temporal “Consistency” Terra: +0.005 before 2004; -.005 after 2004, relative to Aeronet Aqua: no change over time relative to Aeronet From Levy, R., L. Remer, R. Kleidman, S. Mattoo, C. Ichoku, R. Kahn, and T. Eck, 2010. Global evaluation of the Collection 5 MODIS dark-target aerosol products over land, Atmos. Chem. Phys., 10, 10399-10420, doi:10.5194/acp-10-10399-2010.

22. Examining Temporal Representativeness How well does a Daily Level 3 file represent the AOD for that day? Terra Aqua Arctic Subarctic Chance of an overpass during a given hour of the day N. Temperate N. Temperate S. Temperate Subantarctic Antarctic 0 24 0 24 Local Time Local Time

23. How well does a monthly product represent all the days of the month? MODIS Aqua AOD July 2009 MISR Terra AOD July 2009 • Completeness: MODIS dark target algorithm does not work for deserts • Representativeness: monthly aggregation is not enough for MISR and even MODIS • Spatial sampling patterns are different for MODIS Aqua and MISR Terra: “pulsating” areas over ocean are oriented differently due to different orbital direction during day-time measurement

24. Examining Spatial Representativeness Neither pixel count nor standard deviation alone express how representative the grid cell value is

25. Generalizing Accuracy • Aerosol Optical Depth • Different sources of uncertainty: • Low AOD: surface reflectance • High AOD: assumed aerosol models • Also: distribution is closer to lognormal than normal • Thus, “normal” accuracy expressions are problematic: • Slope relative to Ground Truth (Aeronet) • Correlation Coefficient • Root-mean-square error • Instead, common practice with MODIS data is: • Percent falling within expected error bounds ofe.g., +0.05 + 0.2Aeronet

26. Intercomparison of data from multiple sensors Data from multiple sources to be used together: • Current sensors/missions: MODIS, MISR, GOES, OMI. Harmonization needs: • It is not sufficient just to have the data from different sensors and their provenances in one place • Before comparing and fusing data, things need to be harmonized: • Metadata: terminology, standard fields, units, scale • Data: format, grid, spatial and temporal resolution, wavelength, etc. • Provenance: source, assumptions, algorithm, processing steps • Quality: bias, uncertainty, fitness-for-purpose, validation Dangers of easy data access without proper assessment of the joint data usage - It is easy to use data incorrectly

27. Example: South Pacific Anomaly MODIS Level 3 dataday definition leads to artifact in correlation

28. …is caused by an Overpass Time Difference

29. Investigation of artifacts in AOD correlation between MODIS and MISR near the Dateline Using Local time-based data-day definition for each pixel Using calendar data-day definition for each pixel Standard MODIS Terra and MISR Progressively removing artifacts by applying appropriate dataday definition for Level3 daily data generation for both MODIS Terra and MISR

30. Different kinds of reported data quality • Pixel-levelQuality: algorithmic guess at usability of data point • Granule-level Quality: statistical roll-up of Pixel-level Quality • Product-levelQuality: how closely the data represent the actual geophysical state • Record-level Quality: how consistent and reliable the data record is across generations of measurements Different quality types are often erroneously assumed having the same meaning Ensuring Data Quality at these different levels requires different focus and action

31. Sensitivity of Aerosol and Chlorophyll Relationship to Data-Day Definition • The standard Level 3 daily MODIS Aerosol Optical Depth(AOD) at 550nm generated by the atmosphere group uses granule UTC-time based data-day, while the standard Level 3 daily SeaWiFS Chlorophyll (Chl) generated by ocean group uses the pixel-based Local Solar Time (LST) data-day. • The correlation coefficients between Chl and AOT differ significantly near the dateline due to the data-day definition. • This study suggests that the same or similar statistical aggregation methods using the same or similar data-day definitions, should be used when creating climate time series for different parameters to reduce potential emergence of artifacts.

32. Sensitivity Study: Effect of the Data Day definition on Ocean Color data correlation with Aerosol data Starting with Aerosols: Only half of the Data Day artifact is present because the Ocean Group uses the better Data Day definition! Correlation between MODIS Aqua AOD (Ocean group product) and MODIS-Aqua AOD (Atmosphere group product) Pixel Count distribution

33. Sensitivity Study: Effect of the Data Day definition on Ocean Color data correlation with Aerosol data Continuing with Chlorophyll and Aerosols: Data Day effect is quite visible! Pixel Count distribution Correlation between MODIS Aqua Chlorophyll and MODIS-Aqua AOD 550nm (Atmosphere group product) for Apr 1 – Jun 4 2007 GEO-CAPE impact: observation time difference with ACE and other sensors may lead to artifacts in comparison statistics

34. Sensitivity of Aerosol and Chl Relationship to Data-Day Definition Difference between Correlation of A and B: A: MODIS AOT of LST and SeaWiFS Chl B: MODIS AOT of UTC and SeaWiFS Chl Correlation Coefficients MODIS AOT at 550nm and SeaWiFS Chl Artifact: difference between using LST and the calendar UTC-based dataday

35. Presenting data quality to users Split quality (viewed here broadly) into two categories: • Global or product level quality information, e.g. consistency, completeness, etc., that can be presented in a tabular form. • Regional/seasonal: various approaches: • maps with outlines regions, one map per sensor/parameter/season • scatter plots with error estimates, one per a combination of Aeronet station, parameter, and season; with different colors representing different wavelengths, etc.

36. But really what works is…

37. Quality Labels

38. Quality Labels Generated for a request for 20-90 deg N, 0-180 deg E

39. Advisory Report (Dimension Comparison Detail)

40. Advisory Report (Expert Advisories Detail)

41. Quality Comparison Table for Level-3 AOD (Global example)

42. Going down to the individual level

43. The quality of data can vary considerably Version 5 Level 2 Standard Retrieval Statistics

44. Data Quality • Validation of aerosol data show that not all data pixel labeled as “bad” are actually bad if looking at from a bias perspective. • But many pixels are biased due to various reasons From Levy et al, 2009

45. Percent of Biased Data in MODIS Aerosols Over Land Increase as Confidence Flag Decreases *Compliant data are within + 0.05 + 0.2Aeronet Statistics from Hyer, E., J. Reid, and J. Zhang, 2010, An over-land aerosol optical depth data set for data assimilation by filtering, correction, and aggregation of MODIS Collection 5 optical depth retrievals, Atmos. Meas. Tech. Discuss., 3, 4091–4167.

46. The effect of bad qualitydata is often not negligible Hurricane Ike, 9/10/2008 Total Column Precipitable Water Quality Best Good Do Not Use kg/m2

47. …or they can be more complicated Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS) Air Temperatureat 300 mbar PBest : Maximum pressure for which quality value is “Best” in temperature profiles

48. Quality flags are also sometimes packed together into bytes Cloud Mask Status Flag 0=Undetermined1=Determined Day / Night Flag 0=Night1=Day Snow/ Ice Flag 0=Yes1=No Sunglint Flag 0=Yes1=No Cloud Mask Cloudiness Flag 0=Confident cloudy 1=Probably cloudy 2=Probably clear 3=Confident clear Surface Type Flag 0=Ocean, deep lake/river 1=Coast, shallow lake/river 2=Desert 3=Land Bitfield arrangement for the Cloud_Mask_SDS variable in atmospheric products from Moderate Resolution Imaging Spectroradiometer (MODIS)

49. So, replace bad-quality pixels with fill values!!!! Original data array (Total column precipitable water) Mask based on user criteria (Quality level < 2) Good quality data pixels retained Output file has the same format and structure as the input file (except for extra mask and original_data fields)

50. Visualizations help users see the effect of different quality filters Data withinproduct file Best quality only Best + Good quality