Development of Program Level Product Quality Metrics Robert Frouin 1 , Rama Hampapuram 2 , Greg Hunolt 3 , Kamel Didan 4, and others 5 1 Scripps Institution of Oceanography, 2 GSFC / ESDIS, 3 SGT, 4 UofA, 5 MEaSUREs PIs. _________________________________
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Product Quality Metrics
Robert Frouin1, Rama Hampapuram2, Greg Hunolt3, Kamel Didan4, and others5
1Scripps Institution of Oceanography,
2GSFC / ESDIS, 3SGT, 4UofA, 5MEaSUREs PIs
ESDSWG Meeting – MPARWG Breakout 20-22 October 2010, New Orleans
-To measure how well products conform to “requirements” (who and how to define req.?)
-To track maturity and progress (e.g., accuracy and coverage).
-To ascertain whether products are used “properly” (consider user creativity!).
-To take necessary corrective actions or improvements.
-To determine what program level product quality metrics would make sense – i.e. be meaningful, clear and concise, and be practical to collect and report.
-Dimensions and criteria should be defined for programmatic assessments and planning, i.e., they may differ from the detailed standards for product quality developed at the project level.
-From NASA’s viewpoint, the basic standard of information quality has three components: utility, objectivity, and integrity.
-In ensuring the quality of the disseminated NASA “information”, all of these components must be “sufficiently” addressed.
-Utility: Refers to the extent that the information can be used for its intended purpose, by its intended audience.
-Objectivity: Refers to the extent that the information is accurate, clear, complete, and unbiased.
-Integrity: Refers to the protection of NASA’s information from unauthorized access, revision, modification, corruption, falsification, and inadvertent or unintentional destruction.
-The disseminated information and the methods used to produce this information should be as transparentas possible so that they can, in principle, be reproducibleby qualified individuals.
-Accuracy: How does the data agree with independent, correct sources of information (reference data), especially in situ measurements? How biased is the data? How does accuracy depend on spatial and temporal scales, geographic region, and season?
-Consistency: Is the data always produced in the same way (e.g., from one time period to the next)? Is the data coherent spatially and temporally, and does it remain within the expected domain of values? Is the data in accordance with other (relevant) data or information?
-Completeness: Is some data missing (e.g., due to algorithm limitations or nonexistent input)? Is the data sufficiently comprehensive (e.g., long-term, extended spatially) and accurate for usability?
-Relevance: How significant or appropriate is the data for the applications envisioned? What advantages are provided by the data?
-Accessibility: How available, easily and quickly retrievable is the data? Is the data sufficiently up-to-date? Can the data be easily manipulated? Does the data have security restrictions?
-Usabilityis an overarching criteria because for a product to be fully usable the product must not only be of high science quality, but that quality, along with all other information required for use of the product, must be documented.
-This suggests the possibility of defining a set of usability levels that would address not only intrinsic science quality but also the other factors that contribute to, or are required, for a product to be usable (i.e., documentation, accessibility, and support service).
-The usability levels would derive from the science quality, documentation, and accessibility levels, in which criteria defined previously could come into play.
The “Factors” could be selected criteria that apply to Intrinsic Science Quality. Each criterion or ‘factor’ used could have its set of questions, and the answers to those questions could be the basis for “High”, “Medium” or “Low” for that factor.
-In this approach, the metrics associated with usability, intrinsic science quality, documentation, and accessibility / support Services should be defined for those items that need to be tracked at the program level, i.e., that are both important and potentially problematical or a key measure of a project’s process.
-Some level of detail is necessary. Some criteria must be objective, since perceptions of the individuals involved with product development may be subjective.
-The metrics should provide information on the state of the product without the conceptual knowledge of the application (project-independent) and with specific applications in mind (project-dependent).
-The perceived quality of a product by users, or the real world quality of products, may be very different from the analysis by those involved in generating the products.
-User surveys are complementary to internal (i.e., collected from stakeholders) metrics. They are necessary to assess, using comparative analysis, proper usage and adequate documentation and accessibility, which may lead to corrective actions for improving product quality.
Same sensor(s) & a “simple” reprocessing (C4 to C5) leads to major changeSummer NDVI comparisons
Winter EVI comparisons
C4 based Amazon
response to 2005 drought
Amazon response to 2005 drought
Saleska, Didan, Huete & Da Rocha (science 2007)
MODIS C5 EVI based Annual GPP
MODIS C4 EVI based Annual GPP
C5 – C4 Ann. GPP Difference
C5 – C4 Ann. GPP Percent Difference
JFM AMJ JAS OND