DG Joint Research Centre. Formal and informal approaches to the quality of information in integrated assessment Stefano Tarantola January 24-25, 2002 Laxenburg, Austria. Tools for Extended quality assurance. Information used as input to policy-making is complex, uncertain and disputed.
Formal and informal approaches to the quality of information in integrated assessment
January 24-25, 2002 Laxenburg, Austria
Information used as input to policy-making is complex,
uncertain and disputed.
Established guidelines: egNUSAP and model Pedigree schemes for the quality assurance of the decision
Space of the assumptions
Space of the inferencesTo set the frame
[Leamer, 1990 ] (economist):
“I propose a form of organised sensitivity analysis in which a neighborhood of alternative assumptions is selected and the corresponding interval of inferences is identified.
Space of the assumptionsTo set the frame
“Conclusions are judged to be sturdy only if the neighborhood of assumptions is wide enough to be credible and the corresponding interval of inferences is narrow enough to be useful.”
Edward E. Leamer, 1990 “Sensitivity Analysis would help”, in Modelling Economic Series, Edited by CWJ Granger, Clarendon Press, Oxford. Chair in Man. Bus. Econ., UCLA
(the input factors) in the space
of the assumptions (or input space).
uncertaintyTo set the frame
We move one step further: after characterising the interval of inferences (using e.g. the statistical variance),
Input factors should be
interpreted in sensu lato:
- alternative assumptions,
- choice of model,
- algorithmic alternatives,
- poorly-known data...
the interval of the
is too wide
No defensible choice is
possible given the uncertainties.
Space of the assumptions
To validate or invalidate assessments
GSA used to show that the uncertainty in the decision on
whether to burn or dispose solid waste depends on the choice
of the system of indicators, and not on the quality of the
Money should not be spent to improve quality in data, but
to reach a consensus on the proper system of indicators.
Tarantola et al., in Saltelli et al. Eds, (2000)Sensitivity Analysis John Wiley
Problem simplification and dialogue optimisation
We look for those uncertain factors that have negligible
influence on the output.
These can be fixed to the most plausible value within
their range of variation.
The dimensionality of the input space is then reduced.
Useless discussing about the use of different architectures to build the composite indicator, when these do not influence the result.
Output uncertainty reduction
Joint use of UA and GSA (iterative procedure).
Perform UA and get a confidence interval for the output
If it is unacceptably large, acquire better knowledge on the
most important factors. Perform UA again to check ...
It the output quality exceeds the requirements,
the specifications on the input quality can be relaxed,
starting from the less important factors.
Crosetto and Tarantola (2001) Int J Geogr Inf Science
 Saltelli, A., K. Chan, M. Scott, Editors, 2000, Sensitivity analysis, John Wiley & Sons publishers, Probability and Statistics series. Saltelli, A., Chan, K., Scott, M. Eds., 1999, Special Issue on sensitivity analysis, Computer Physics Communications, 117.  Saltelli A., Tarantola S., and Chan K., 1999, A quantitative, model independent method for global sensitivity analysis of model output, Technometrics, 41(1), 39-56.  Saltelli A., Tarantola S., Campolongo F., 2001, Sensitivity analysis as an ingredient of modelling, Statistical Science, 15(4), 377-395.