Peter Fox (TWC/RPI), and … Stephan Zednik 1 , Gregory Leptoukh 2 ,

1. Informatics takes on data and information quality, uncertainty and bias (in atmospheric science) Peter Fox (TWC/RPI), and … Stephan Zednik1, Gregory Leptoukh2, Chris Lynnes2, Jianfu Pan3 Tetherless World Constellation, Rensselaer Polytechnic Inst. NASA Goddard Space Flight Center, Greenbelt, MD, United States Adnet Systems, Inc.

2. Where are we in respect to this data challenge? “The user cannot find the data; If he can find it, cannot access it; If he can access it, ; he doesn't know how good they are; if he finds them good, he can not merge them with other data” The Users View of IT, NAS 1989

3. Definitions (ATM) • Quality • Is in the eyes of the beholder – worst case scenario… or a good challenge • Uncertainty • has aspects of accuracy (how accurately the real world situation is assessed, it also includes bias) and precision (down to how many digits) • Bias has two aspects: • Systematic error resulting in the distortion of measurement data caused by prejudice or faulty measurement technique • A vested interest, or strongly held paradigm or condition that may skew the results of sampling, measuring, or reporting the findings of a quality assessment: • Psychological: for example, when data providers audit their own data, they usually have a bias to overstate its quality. • Sampling: Sampling procedures that result in a sample that is not truly representative of the population sampled. (Larry English)

4. Data quality needs: fitness for purpose/ use • Measuring Climate Change: • Model validation: gridded contiguous data with uncertainties • Long-term time series: bias assessment is the must , especially sensor degradation, orbit and spatial sampling change • Studying phenomena using multi-sensor data: • Cross-sensor bias characterization is needed • Realizing Societal Benefits through Applications: • Near-Real Time for transport/event monitoring - in some cases, coverage and timeliness might be more important that accuracy • Pollution monitoring (e.g., air quality exceedance levels) – accuracy • Educational (users generally not well-versed in the intricacies of quality; just taking all the data as usable can impair educational lessons) – only the best products

5. Same parameter Same space & time MODIS vs. MERIS MODIS MERIS Different results – why? A threshold used in MERIS processing effectively excludes high aerosol values. Note: MERIS was designed primarily as an ocean-color instrument, so aerosols are “obstacles” not signal.

6. Spatial and temporal sampling – how to quantify to make it useful for modelers? • MODIS Aqua AOD July 2009 • MISR Terra AOD July 2009 • Completeness: MODIS dark target algorithm does not work for deserts • Representativeness: monthly aggregation is not enough for MISR and even MODIS • Spatial sampling patterns are different for MODIS Aqua and MISR Terra: “pulsating” areas over ocean are oriented differently due to different orbital direction during day-time measurement  Cognitive bias

7. Anomaly Example: South Pacific Anomaly Anomaly MODIS Level 3 dataday definition leads to artifact in correlation

8. …is caused by an Overpass Time Difference

9. Sensitivity Study: Effect of the Data Day definition on Ocean Color data correlation with Aerosol data Only half of the Data Day artifact is present because the Ocean Group uses the better Data Day definition! Correlation between MODIS Aqua AOD (Ocean group product) and MODIS-Aqua AOD (Atmosphere group product) Pixel Count distribution

10. Why so difficult? • Quality is perceived differently by data providers and data recipients. • There are many different qualitative and quantitative aspects of quality. • Methodologies for dealing with data qualities are just emerging • Almost nothing exists for remote sensing data quality • Even the most comprehensive review (Batini’s book) demonstrates that there are no preferred methodologies for solving many data quality issues • Little funding was allocated in the past to data quality as the priority was to build an instrument, launch a rocket, collect and process data, and publish a paper using just one set of data. • Each science team handled quality differently.

11. More terminology • ‘Even a slight difference in terminology can lead to significant differences between data from different sensors. It gives an IMPRESSION of data being of bad quality while in fact they measure different things. For example, MODIS and MISR definitions of the aerosol "fine mode" is different, so the direct comparison of fine modes from MODIS and MISR does not always give good correlation.’ • Ralph Kahn, MISR Aerosol Lead.

12. Quality Control vs. Quality Assessment Quality Control (QC) flags in the data (assigned by the algorithm) reflect “happiness” of the retrieval algorithm, e.g., all the necessary channels indeed had data, not too many clouds, the algorithm has converged to a solution, etc. Quality assessment is done by analyzing the data “after the fact” through validation, intercomparison with other measurements, self-consistency, etc. It is presented as bias and uncertainty. It is rather inconsistent and can be found in papers, validation reports all over the place.

13. Different kinds of reported data quality • Pixel-levelQuality: algorithmic guess at usability of data point • Granule-level Quality: statistical roll-up of Pixel-level Quality • Product-levelQuality: how closely the data represent the actual geophysical state • Record-level Quality: how consistent and reliable the data record is across generations of measurements Different quality types are often erroneously assumed having the same meaning Ensuring Data Quality at these different levels requires different focus and action

14. Three projects with data & information quality flavor • Multi-sensor Data Synergy Advisor (**) • Product level • Goal: Provide science users with clear, cogent information on salient differences between data candidates for fusion, merging and intercomparison and enable scientifically and statistically valid conclusions • Develop MDSA on current missions – Terra, Aqua, (maybe Aura) • Define implications for future missions • Data Quality Screening Service • Pixel level • Aerosol Status • Record level

15. Giovanni Earth Science Data Visualization & Analysis Tool • Developed and hosted by NASA/ Goddard Space Flight Center (GSFC) • Multi-sensor and model data analysis and visualization online tool • Supports dozens of visualization types • Generate dataset comparisons • ~1500 Parameters • Used by modelers, researchers, policy makers, students, teachers, etc.

16. Giovanni Allows Scientists to Concentrate on the Science Exploration Initial Analysis Use the best data for the final analysis Derive conclusions Write the paper Submit the paper The Old Way: The Giovanni Way: Web-based Services: Jan Pre-Science Find data Minutes Retrieve high volume data Read Data Feb Extract Parameter Learn formats and develop readers Days for exploration Filter Quality Mirador Extractparameters Mar Use the best data for the final analysis Subset Spatially Giovanni Perform spatial and other subsetting DO SCIENCE Derive conclusions Reformat Apr Identify quality and other flags and constraints Write the paper Reproject Submit the paper Perform filtering/masking Visualize May Develop analysis and visualization Explore Accept/discard/get more data (sat, model, ground-based) Analyze Jun Web-based tools like Giovanni allow scientists to compress the time needed for pre-science preliminary tasks: data discovery, access, manipulation, visualization, and basic statistical analysis. Jul DO SCIENCE Aug Sep Scientists have more time to do science! Oct

17. Data Usage Workflow

18. Data Usage Workflow *Giovanni helps streamline / automate Subset / Constrain Reformat Filtering Re-project Integration

19. Data Usage Workflow Precision Requirements *Giovanni helps streamline / automate Integration Planning Quality Assessment Requirements Intended Use Subset / Constrain Reformat Filtering Re-project Integration

20. Informatics approach • Systematizing quality aspects • Working through literature • Identifying aspects of quality and their dependence of measurement and environmental conditions • Developing Data Quality ontologies • Understanding and collecting internal and external provenance • Developing rulesets allows to infer pieces of knowledge to extract and assemble • Presenting the data quality knowledge with good visual, statement and references

21. Data Quality Ontology Development (Quality flag) Working together with Chris Lynnes’s DQSS project, started from the pixel-level quality view.

22. Data Quality Ontology Development (Bias) http://cmapspublic3.ihmc.us:80/servlet/SBReadResourceServlet?rid=1286316097170_183793435_22228&partName=htmltext

23. Modeling quality (Uncertainty) Link to other cmap presentations of quality ontology: http://cmapspublic3.ihmc.us:80/servlet/SBReadResourceServlet?rid=1299017667444_1897825847_19570&partName=htmltext

24. MDSA Aerosol Data Ontology Example Ontology of Aerosol Data made with cmap ontology editor

25. RuleSet Development [DiffNEQCT: (?s rdf:type gio:RequestedService), (?s gio:input ?a), (?a rdf:type gio:DataSelection), (?s gio:input ?b), (?b rdf:type gio:DataSelection), (?a gio:sourceDataset ?a.ds), (?b gio:sourceDataset ?b.ds), (?a.ds gio:fromDeployment ?a.dply), (?b.ds gio:fromDeployment ?b.dply), (?a.dply rdf:type gio:SunSynchronousOrbitalDeployment), (?b.dply rdf:type gio:SunSynchronousOrbitalDeployment), (?a.dply gio:hasNominalEquatorialCrossingTime ?a.neqct), (?b.dply gio:hasNominalEquatorialCrossingTime ?b.neqct), notEqual(?a.neqct, ?b.neqct) -> (?s gio:issueAdvisory giodata:DifferentNEQCTAdvisory)]

26. Assisting in Assessment Precision Requirements Quality Assessment Requirements Integration Planning Provenance & Lineage Visualization Intended Use MDSA Advisory Report Subset / Constrain Reformat Filtering Re-project Integration

27. Advisor Knowledge Base Advisor Rules test for potential anomalies, create association between service metadata andanomaly metadata in Advisor KB

28. Semantic Advisor Architecture RPI

29. Advisory Report (Dimension Comparison Detail)

30. Advisory Report (Expert Advisories Detail)

31. Summary • Quality is very hard to characterize, different groups will focus on different and inconsistent measures of quality • Modern ontology representations to the rescue! • Products with known Quality (whether good or bad quality) are more valuable than products with unknown Quality. • Known quality helps you correctly assess fitness-for-use • Harmonization of data quality is even more difficult that characterizing quality of a single data product

32. Summary • Advisory Report is not a replacement for proper analysis planning • But provides benefit for all user types summarizing general fitness-for-usage, integrability, and data usage caveat information • Science user feedback has been very positive • Provenance trace dumps are difficult to read, especially to non-software engineers • Science user feedback; “Too much information in provenance lineage, I need a simplified abstraction/view” • Transparency  Translucency • make the important stuff stand out

33. Future Work • Advisor suggestions to correct for potential anomalies • Views/abstractions of provenance based on specific user group requirements • Continued iteration on visualization tools based on user requirements • Present a comparability index / research techniques to quantify comparability

34. Extra material

35. Acronyms ACCESS Advancing Collaborative Connections for Earth System Science ACE Aerosol-Cloud-Ecosystems AGU American Geophysical Union AIST Advanced Information Systems Technology AOD Aerosol Optical Depth AVHRR Advanced Very High Resolution Radiometer GACM Global Atmospheric Composition Mission GeoCAPE Geostationary Coastal and Air Pollution Events GEWEX Global Energy and Water Cycle Experiment GOES Geostationary Operational Environmental Satellite GOME-2 Global Ozone Monitoring Experiment-2 JPSS Joint Polar Satellite System LST Local Solar Time MDSA Multi-sensor Data Synergy Advisor MISR Multiangle Imaging SpectroRadiometer MODIS Moderate Resolution Imaging Spectraradiometer NPP National Polar-Orbiting Operational Environmental Satellite System Preparatory Project

36. Acronyms (cont.) OMI Ozone Monitoring Instrument OWL Web Ontology Language PML Proof Markup Language QA4EO QA for Earth Observations REST Representational State Transfer TRL Technology Readiness Level UTC Coordinated Universal Time WADL Web Application Description Language XML eXtensible Markup Language XSLeXtensibleStylesheet Language XSLTXSL Transformation

37. Quality & Bias assessment using FreeMind FreeMind allows capturing various relations between various aspects of aerosol measurements, algorithms, conditions, validation, etc. The “traditional” worksheets do not support complex multi-dimensional nature of the task from the Aerosol Parameter Ontology

38. Title:MODISTerraC5AOD vs. Aeronet during Aug-Oct Biomass burningin Central Brazil, South America (General) Statement: Collection 5 MODIS AOD at 550 nm during Aug-Oct over Central South America highly over-estimates for large AOD and in non-burning season underestimates for small AOD, as compared to Aeronet; good comparisons are found at moderate AOD. Region & season characteristics: Central region of Brazil is mix of forest, cerrado, and pasture andknown to have low AOD most of the year except during biomass burning season * Alta Floresta * MatoGrosso • (Dominating factors leading to Aerosol Estimate bias): • Large positive bias in AOD estimate during biomass burning season may be due to wrong assignment of Aerosol absorbing characteristics.(Specific explanation) a constant Single Scattering Albedo ~ 0.91 is assigned for all seasons, while the true value is closer to ~0.92-0.93. • [ Notes or exceptions: Biomass burning regions in Southern Africa do not show as large positive bias as in this case, it may be due to different optical characteristics or single scattering albedo of smoke particles, Aeronet observations of SSA confirm this] • 2. Low AOD is common in non burning season. In Low AOD cases, biases are highly dependent on lower boundary conditions. In general a negative bias is found due to uncertainty in Surface Reflectance Characterization which dominates if signal from atmospheric aerosol is low. * Santa Cruz Central South America 2 MODIS AOD 1 (Example) : Scatter plot of MODIS AOD and AOD at 550 nm vs. Aeronet from ref. (Hyer et al, 2011) (Description Caption) shows severe over-estimationof MODIS Col 5 AOD (dark target algorithm) at large AOD at 550 nm during Aug-Oct 2005-2008 over Brazil. (Constraints) Only best quality of MODIS data (Quality =3 ) used. Data with scattering angle > 170 deg excluded. (Symbols) Red Lines define regions of Expected Error (EE), Green is the fitted slope Results: Tolerance= 62% within EE; RMSE=0.212 ; r2=0.81; Slope=1.00 For Low AOD (<0.2) Slope=0.3. For high AOD (> 1.4) Slope=1.54 0 1 2 Aeronet AOD Reference:Hyer, E. J., Reid, J. S., and Zhang, J., 2011: An over-land aerosol optical depth data set for data assimilation by filtering, correction, and aggregation of MODIS Collection 5 optical depth retrievals, Atmos. Meas. Tech., 4, 379-408, doi:10.5194/amt-4-379-2011