Building composite indices – methodology and quality issues A. Saltelli

1. Building composite indices – methodology and quality issues A. Saltelli European Commission, Joint Research Centre of Ispra, Italy andrea.saltelli@jrc.it Indicators and Assessment WorkshopEEA & USEPA/OEI joint meeting, September 27th – 28th, 2004, Brussels

2. Prepared with Michaela Saisana and Stefano Tarantola – based on: Saisana M., Saltelli A., Tarantola S. (2005) Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, J. R. Stat. Soc. A, 168, Part 2, pp.1-17. Forthcoming Joint OECD JRC handbook on good practices in composite indictors building

3. http://farmweb.jrc.cec.eu.int/ci/

4. Outline • CI controversy • Composite Indicators as models • Wackernagel’s critique of ESI … • Putting the critique into practice: the TAI example • Conclusions

5. CI controversy • EU structural indicators: • scoreboard versus index

6. Report from the Commission to the Spring European Council 2004, Annex 1 Relative Performance Relative Improvement in Performance (av. since 1999)

7. Assessing policies: Green – Country policy on a good path; Yellow – Country policy on a bad path (expert judgment)

8. The Commission also provided the Secretary General with an aggregated analysis table

9. Aggregated analysis of relative performance • Good - Countries on a good path Number of (good cells - bad cells)≥ 3 • Poor - Countries on a bad path Number of (bad cells - good cells) ≥ 3 • Average All others

10. Enter the FT analysts … Source: Spring Report, European Commission 2004 Source: Financial Times Thursday January 22 2004

11. Categorisation (star rating[*]) in three groups • LEADERSUK, NL SE, DK, AT,LU • MIDDLE OF THE ROADDE, FI, IE, BE, FR • LAGGARDSIT, GR, ES, PT • done by FT and based likely on same synoptic performance and improvement tables in the Spring Report, 2004, Annex 1 (yellow-green boxes) • [*]Like in the UK NHS hospital rating

12. Can “league tables” be avoided? Or are they an ingredient of an overall analysis and presentational strategy: Long list of 107 Short List of 14 Synoptic tables League tables

13. <<The aggregators believe there are two major reasons that there is value in combining indicators in some manner to produce a bottom line. They believe that such a summary statistic can indeed capture reality and is meaningful, and that stressing the bottom line is extremely useful in garnering media interest and hence the attention of policy makers. The second school, the non-aggregators, believe one should stop once an appropriate set of indicators has been created and not go the further step of producing a composite index. Their key objection to aggregation is what they see as the arbitrary nature of the weighting process by which the variables are combined.>> • “Literature Review of Frameworks for Macro-indicators”, • Andrew Sharpe, 2004, Centre for the Study of Living Standards, Ottawa, CAN.

14. Reviews on methodologies and practices on composite indicators : State-of-the-art Report on Current Methodologies and Practices for Composite Indicator Development (2002) Michaela Saisana & Stefano Tarantola, European Commission, Joint Research Centre Composite indicators of country performance: a critical assessment (2003) Michael Freudenberg, OECD. Literature Review of Frameworks for Macro-indicators (2004), Andrew Sharpe, Centre for the Study of Living Standards, Ottawa, CAN. Measuring performance: An examination of composite performance indicators (2004) Rowena Jacobs, Peter Smith, Maria Goddard, Centre for Health Economics, University of York, UK. Methodological Issues Encountered in the Construction of Indices of Economic and Social Well-being (2003) Andrew Sharpe Julia Salzman Methodological Choices Encountered in the Construction of Composite Indices of Economic and Social Well-Being, Julia Salzman , (2004) Center for the Study of Living Standards , Ottawa, CAN. http://farmweb.jrc.cec.eu.int/ci/

15. Pros & Cons (Saisana and Tarantola, 2002) • Pros • Composite indicators can be used to summarise complex or multi-dimensional issues, in view of supporting decision-makers. • Composite indicators provide the big picture. They can be easier to interpret than trying to find a trend in many separate indicators. They facilitate the task of ranking countries on complex issues. • Composite indicators can help attracting public interest by providing a summary figure with which to compare the performance across countries and their progress over time. • Composite indicators could help to reduce the size of a list of indicators or to include more information within the existing size limit.

16. Pros & Cons (EC, 2002) • Cons • Composite indicators may send misleading, non-robust policy messages if they are poorly constructed or misinterpreted. Sensitivity analysis can be used to test composite indicators for robustness. • The simple “big picture” results which composite indicators show may invite politicians to draw simplistic policy conclusions. Composite indicators should be used in combination with the sub-indicators to draw sophisticated policy conclusions. • The construction of composite indicators involves stages where judgement has to be made: the selection of sub-indicators, choice of model, weighting indicators and treatment of missing values etc. These judgements should be transparent and based on sound statistical principles. • There could be more scope for disagreement among Member States about composite indicators than on individual indicators. The selection of sub-indicators and weights could be the target of political challenge • The composite indicators increase the quantity of data needed because data are required for all the sub-indicators and for a statistically significant analysis.

17. Pros & Cons (JRSS paper) “[…] it is hard to imagine that debate on the use of composite indicators will ever be settled[…] official statisticians may tend to resent composite indicators, whereby a lot of work in data collection and editing is “wasted” or “hidden” behind a single number of dubious significance. On the other hand, the temptation of stakeholders and practitioners to summarise complex and sometime elusive processes (e.g. sustainability, single market policy, etc.) into a single figure to benchmark country performance for policy consumption seems likewise irresistible.”

18. Composite indicators as models … and the critique of models

19. Indicators as models … and the critique of models The nature of models, after Rosen

20. Decoding F Formal system N Natural system Entailment Entailment Encoding The critique of models After Rosen, 1991, ”World” (the natural system) and “Model” (the formal system) are internally entailed - driven by a causal structure. [Efficient, material, final for ‘world’ – formal for ‘model’]Nothing entails with one another “World” and “Model”; the association is hence the result of a craftsmanship.

21. Wackernagel’s critique of ESI …

22. The critique of indicators Environmental sustainability Index, figure from The Economist, Green and growing, The Economist, Jan 25th 2001, Produced on behalf of the World Economic Forum (WEF), and presented to the annual Davos summit this year.

23. The critique of indicators: Robustness … Mathis Wackernagel, mental father of the “Ecological Footprint” and thus an authoritative source in the Sustainable Development expert community, concludes an argumented critique of the study done presented at Davos by noting:

24. "Overall, the report would gain from a more extensive peer review and a sensitivity analysis. The lacking sensitivity analysis undermines the confidence in the results since small changes in the index architecture or the weighting could dramatically alter the ranking of the nations.” The critique of indicators: Robustness …

25. Putting the critique into practice: the TAI example

26. How to tackle it: the TAI example We have tried to address the issue of robustness taking as example the UN Technology Achievement index (2001), Human Development Report series [An example on the ESI 2005 itself will be ready before the end of the year – draft on 2002 data available]

27. The TAI example • Technology achievement index: 4 dimensions, 8 indicators • Creation of technology • II) Diffusion of recent innovations • III) Diffusion of old innovations • IV) Human skills

28. How to tackle it: the TA example Tools: Uncertainty Analysis (UA) and Sensitivity Analysis (SA). UA focuses on how uncertainty in the input factors propagates through the structure of the composite indicator and affects the composite indicator values. SA studies how much each individual source of uncertainty contributes to the output variance.

29. Uncertainty analysis = Mapping assumptions onto inferences Sensitivity analysis = The reverse process Simplified diagram - fixed model

30. Answers being sought from Tai example: • Does the use of one normalisation method and one set of weights in the development of the composite indicator (e.g. original TAI) provide a biased picture of the countries’ performance? • To what extent do the uncertain input factors (normalisation methods, weighting schemes and weights) affect the countries’ ranks with respect to the original TAI?

31. Set up of the analysis - Weighting schemes Two participatory methods: Budget allocation (experts allocate a finite number of points among the set of indicators) or Analytichierarchy process (expert compare the indicators pairwise, and express numerically the relative importance on a 1 [indifference]-9 [much more important] scale. Possibility of inconsistencies.

32. Propagation of Uncertainty Steps in building a composite indicator Inputs Model Output Values of Technology Achievement Index for different countries 1. selection of sub-indicators 2. data selection 3. data editing 4. data normalisation 5. weighting scheme 6. weights’ values 7. composite indicator formula e.g. in its simplest form: xi wi : indicators : weights

33. Uncertainty analysis – Results for The values of the composite indicator are displayed in the form of confidence bounds Blue – original TAI Red – median of Monte Carlo TAI

34. Uncertainty analysis – Results for Significant difference TAI – median of MonteCarloTAI A few countries show significant overlap and therefore the ranking is unclear Blue – original TAI Red – Monte Carlo TAI

35. Uncertainty analysis – Results for Answer to question 1 For most countries, the original TAI value is very close to the MC-TAI median value. This implies that the original TAI (one normalisation method and one set of equal weights), provides a picture of the countries’ technological achievements that is not generally biased. (Exception: Netherlands - Singapore)

36. Uncertainty Analysis for In cases where partial overlapping between two countries occurs, the difference in the TAI values for that pair of countries can be further analyzed in via sensitivity analysis 65% of the area

37. Uncertainty Analysis Sensitivity analysis ↓ Moving from uncertainty to sensitivity analysis Each slice is the fractional contribution of a factor [or group of factors] to variance of the output

38. Sensitivity analysis - which uncertain input affects the difference Singapore – Netherlands? Fixing non influential factors

39. Which uncertain input affects the difference Singapore – Netherlands? Further analysis (e.g. scatter-plots) reveals that MC-TAI favours Singapore when high weights are assigned to Enrolment sub-indicator, for which Singapore is much better than the Netherlands, and/or to Electricity sub-indicator, for which Singapore is marginally better than the Netherlands.

41. Inference from sensitivity analysis for the difference Singapore – Netherlands? (Question 2) • The weights for Electricity and Enrolment and the type of weighting approach (BA or AHP) are important • The selection of the normalisation method does not affect the output variance. • The trigger for the weighting scheme has a strong interaction with other factors, mainly with the weights for Electricity, Enrolment and Exports. • …

42. Conclusions • The analysis was useful in showing that: • The original TAI was not generally biased. • Exceptions (Netherlands and Singapore) can be identified and explained = identifying regions in the space of the weights that favour one country with respect to another  input for iterative CI building • The weighting approach matters, normalisation does not  input for iterative CI building