Aggregating Multidimensional Indicators in Europe

1. Jean Monnet Events - 2 May 2019 Aggregating Multidimensional Indicators in Europe Vincenzo Mauro Department of Economics and management University of Pisa

2. Aim of the presentation The aim of this paper is to briefly introduce some of the main issues of multidimensional indexes and their development processes A composite indicator is formed when Indicators are compiled into a single index, on the basis of an underlying model of the multi-dimensional concept that is being measured

3. Premises • Synthesis as a way to measure multidimensional phenomena • Complexity, multidimensionality vs usability • From unidimensional, to multidimensionality…and back • Monitoring outcomes e.g. 17 SDGs and sub-Indicators/targets, including effectiveness • Links between micro, meso and macro • The Capability Approach and Sustainable Human Development  HDI, MPI

4. Outline of the presentation Premises Introduction 10 (and more…) steps to build a composite index Normalisation and Substitutability between indicators Some examples (MPI, HDI, SDGs) Conclusions

5. Background • Which is the class of synthesis of indicator to be used? • Synthesis as a tool to measure, summarise, and rank observations, usually based on multiple data items • In the context introduced, it is usually a function • I:XR • where X is the data matrix with generic entry xijrepresenting the j-th achievement for individual i

6. Background Horizontal and vertical aggregation Data matrix

7. Background Horizontal and vertical aggregation Within-unit aggregation

8. Background Horizontal and vertical aggregation Between-unit aggregation

9. Background Horizontal and vertical aggregation

10. Background Horizontal and vertical aggregation

11. Background Horizontal and vertical aggregation Within-unit aggregation (horizontal) Between-unit aggregation (vertical) These two phases are very different and should be kept separated (e.g. inequality) In this presentation we focus on horizontal aggregation only

12. Background • Common distinction: • Counting measures (e.g. MPI) • Index measures (e.g. HDI) • Formal distinction? They are both functions from the set of Xnxk matrices to a real value • Central role of the underlying assumptions (sometimes not very transparent)

13. From desirable properties of the synthesis to the function These assumptions sometimes appear more as inevitable consequences of the methodology chosen than the result of philosophical and theoretically sound considerations. Assumptions (properties) Function (index)

14. What are the main properties? • Main properties • Full sensitivity of the synthesis to any change in the data for any subgroup and in any dimension (strict monotonicity required for monitoring) • Continuity • A straightforward interpretation of the obtained synthetic score (not only through a comparison) • A theoretically-coherent structure of substitutability between achievements, based on theoretical considerations

15. 10-Step guide for the construction of a composite indicator A good starting point is the 2008 OECD/JRC Handbook on how to build a composite indicator. It suggest 10 steps to follow It represents a ‘decalogue’ for the construction of a composite indicator, which has been rearranged and extended from the information contained in the 2008 OECD/JRC Handbook.

16. This short guide stresses the importance of conducting an internal coherence assessment prior to the uncertainty and sensitivity analysis, so as to further refine and eventually correct the composite indicator structure. Expert opinion is needed in this phase in order to assess the results of the statistical analysis. Second, it stresses that there is a trade-off between multidimensionality and robustness in a composite indicator. One could have a very robust yet mono-dimensional index or a very volatile yet multi-dimensional one.

17. This does not imply that the first index is better than the second one. Treating robustness analysis NOT as an attribute of a composite indicator but of the inference which the composite indicator has been called upon to support. It also highlights the iterative nature of the ten steps, which although presented consecutively in the handbook, the benefit to the developer is in the iterative nature of the steps.

18. Step 1. Theoretical/Conceptual frameworkprovides the basis for the selection and combination of variables into a meaningful composite indicator under a fitness-for-purpose principle (involvement of experts and stakeholders is important). - Clear understanding and definition of the multidimensional phenomenon to be measured. - Discuss the added-value of the composite indicator. - Nested structure of the various sub-groups of the phenomenon (if relevant). - List of selection criteria for the underlying variables, e.g., input, output, process.

19. Step 2. Data selectionshould be based on the analytical soundness, measurability, country coverage, and relevance of the indicators to the phenomenon being measured and relationship to each other. The use of proxy variables should be considered when data are scarce (involvement of experts and stakeholders is important). - Quality assessment of the available indicators. - Discuss strengths and weaknesses of each selected indicator. - Summary table on data characteristics, e.g., availability (across country, time), source, type (hard, soft or input, output, process), descriptive statistics (mean, median, skewness, kurtosis, min, max, variance, histogram).

20. Step 3. Data treatmentconsists of imputing missing data, (eventually) treating outliers and/or making scale adjustments. - Confidence interval for each imputed value that allows assessing the impact of imputation on the composite indicator results. - Discuss and treat outliers, so as to avoid that they become unintended benchmarks (e.g., by applying Box-Cox transformations such square roots, logarithms, and other). - Make scale adjustments, if necessary (e.g., taking logarithms of some indicators, so that differences at the lower levels matter more).(back to step 2)

21. Step 4. Multivariate analysisshould be used to study the overall structure of the dataset, assess its suitability, and guide subsequent methodological choices (e.g., weighting, aggregation). - Assess the statistical and conceptual coherence in the structure of the dataset (e.g., by principal component analysis and correlation analysis). - Identify peer groups of countries based on the individual indicators and other auxiliary variables (e.g., by cluster analysis).(back to Step 1 and Step 2)

22. Step 5. NormalisationCrucial step that should be carried out to render the variables comparable. - Make directional adjustment, so that higher values correspond to better performance in all indicators (or vice versa). - Select a suitable normalisation method (e.g., min-max, z-scores, and distance to best performer) that respects the conceptual framework and the data properties.

23. Normalisation issues Implicit weighting issue • For the standardization, in order to avoid implicit weighting issues, the minimum and maximum bounds are not data-driven but set theoretically for each dimensions  This allows for space and time comparability  Democratic processes, public reasoning (choice and weights, impartial spectator)  Alignment techniques to be used

24. Step 6. Weighting and aggregationshould be done along the lines of the theoretical/conceptual framework - Discuss whether compensability among indicators should be allowed and up to which level of aggregation. - Discuss whether correlation among indicators should be taken into account during the assignment of weights. - Select a suitable weighting and aggregation method that respect the conceptual framework and the data properties. Popular weighting methods include equal weights, factor analysis derived weights, expert opinion, and data envelopment analysis. Popular aggregation methods include arithmetic average, geometric average, Borda, Copeland.

25. Recent developments: the substitutability between dimensions «…there is an inescapable arbitrariness in the choice of the order α…» (Anand and Sen, 1997)

26. Substitutability between dimensions(a simple example) There is an inescapable arbitrariness in the choice of the order α (Anand and Sen, 1997)

27. A possible escape from arbitrariness? Figure: Geometric representation of the new synthesis of indicators

28. Internal coherence assessment (intermediate step)This step is briefly listed under step 9 in the Handbook but not thoroughly discussed. This assessment needs to be undertaken prior to the uncertainty and sensitivity analysis, so as to further refine the composite indicator structure (upon consultation with experts on the issue).-Assess whether dominance problems are present, namely the composite indicator results are overly dominated by a small number of indicators and quantify the relative importance of the underlying components (e.g., by global sensitivity analysis, correlation ratios). - Assess eventual “noise” added to the final composite indicator results by non-influential indicators. - Assess the direction of impact of indicators and sub-dimensions, namely whether all components point to the same direction as the composite indicator (sign of correlation) and explain trade-offs.

29. - Assess whether certain indicators are statistically grouped under different dimensions than conceptualised and whether certain dimensions should be merged or split. - Assess eventual bias introduced in the index (e.g., due to population size, population density)(back to Step 1 and Step 2)

30. Step 7. Uncertainty and sensitivity analysisshould be undertaken to assess the robustness of the composite indicator scores/ranks to the underlying assumptions and to identify which assumptions are more crucial in determining the final classification. The trade-off between multidimensionality and robustness in a composite indicator, given that a mono-dimensional index is likely to be more robust than a multi-dimensional one. This does not imply that the first index is better than the second one. In fact, robustness analysis should NOT be treated as an attribute of the composite indicator but of the inference which the composite indicator has been called upon to support.

31. - Consider different methodological paths to build the index, and if available, different conceptual frameworks. - Identify the sources of uncertainty underlying in thedevelopment of the composite indicator and provide the composite scores/ranks with confidence intervals. - Explain why certain countries notably improve or deteriorate their relative position given the assumptions. - Conduct sensitivity analysis to show what sources of uncertainty are more influential in determining the scores/ranks.

32. Step 8. Relation to other indicatorsshould be made to correlate the composite indicator (or its dimensions) with existing (simple or composite) indicators and to identify linkages. - Correlate the composite indicator with relevant measurable phenomena and explain similarities or differences. - Develop data-driven narratives on the results. - Perform causality tests (if time series data are available).

33. Step 9. Decomposition into the underlying indicatorsshould be carried out to reveal drivers for good/bad performance. - Profile country performance at the indicator level to reveal strengths and limitations. - Perform causality tests (if time series data are available).

34. Step 10. Visualisation of the resultsshould receive proper attention given that it can influence (or help to enhance) interpretability. - Identify suitable presentational tools for the targeted audience. - Select the visualisation technique which communicates the most information without hiding vital information. - Present the results in a clear, easy to grasp and accurate manner.

35. Nature of the relationship • Nature of the relationship between the defined concept and the selected indicators • Reflective vs Formative • Reflectiveindicators are seen as functions of the conceptual (latent) variable (highly correlated and interchangeable) • On the opposite, formative indicators are assumed as causes of the latent variable, so that they are not necessarily correlated to each other

36. Nature of the relationship between the defined concept and the selected indicators • Reflective vs Formative • Two uncorrelated indicators can both contribute to the measurement of the same conceptual variable, while two correlated indicators may turn out to be redundant in measuring the concept

38. The Multidimensional Poverty Index: a brief overview The Global Multidimensional Poverty Index (MPI) was developed in 2010 by the Oxford Poverty & Human Development Initiative and the United Nations Development Programme and uses different factors to determine poverty beyond income-based lists. It replaced the previous Human Poverty Index. The global MPI is released annually by OPHI and the results published on its website.

39. The Multidimensional Poverty Index: a brief overview overview of the multidimensional measurement methodology of Alkire and Foster (2007, 2011a), with an emphasis on the first measure of that class: the Headcount Ratio H

40. The MPI: overview 1) Defining the set of indicators which will be considered in the multidimensional measure. Data for all indicators need to be available for the same person. 2) Setting the deprivation cutoffs for each indicator, namely the level of achievement considered sufficient (normatively) in order to be non-deprived in each indicator. 3) Applying the cutoffs to ascertain whether each person is deprived or not in each indicator.

41. The MPI: overview 4) Selecting the relative weight or value that each indicator has, such that these sum to one (let’s assume they are all equals) 5) Creating the (weighted) sum of deprivations for each person, which can be called his or her ‘deprivation score’. 6) Determining (normatively) the poverty cutoff, namely, the proportion of weighted deprivations a person needs to experience in order to be considered multidimensionally poor, and identifying each person as multidimensionally poor or not according to the selected poverty cutoff.

42. Background The MPI: overview of strengths and limits MPI strengths Positive aspects intuitive poverty focused quite simple usable with ordinal data … but also some limits related to

43. MPI limits in monitoring • The index is not sensitive to changes in the level but on change in the poverty status This does not allow during monitoring of the SGDs to capture the changes for those who are v. poor (and also those who are v. rich), e.g. it works for those close to the thresholds and those crossing it The index does not take into account the heterogeneity between the achievement in each dimensions The MPI is robust but at expense of sensitivity e.g. monitoring of the single person for single dimension and at aggregate level  Index measures are usually better in these aspects

44. SDGs • The Sustainable Development Goals (SDGs) are a collection of 17 global goals set by the United Nations General Assembly in 2015 for the year 2030. The SDGs are part of the 2030 Agenda

45. The Sustainable Development Goals are: 1) No Poverty, 2) Zero Hunger, 3) Good Health and Well-being, 4) Quality Education, 5) Gender Equality, 6) Clean Water and Sanitation, 7) Affordable and Clean Energy, 8) Decent Work and Economic Growth, 9) Industry, Innovation, and Infrastructure, 10) Reducing Inequality, 11) Sustainable Cities and Communities, 12) Responsible Consumption and Production, 13) Climate Action, 14) Life Below Water, 15) Life On Land, 16) Peace, Justice, and Strong Institutions, 17) Partnerships for the Goals.The goals are broad based and interdependent. The 17 Sustainable Development Goal's each have a list of targets that are measured with indicators.

46. To date only one contribution attempts to develop and apply a single unified indicator for monitoring progress towards the SDGs at the global level. The indicator in question is theSustainable Development Goal Index (SDG-I), that has been developed by Jeffrey Sachs et al. (2016; 2017; 2018)

47. It draws on available data from a variety of publicly available sources for all 193-member states of the United Nations from the year 2016 onwards. The SDG-I is derived from a scoring system that uses the arithmetic mean to aggregate indicators relating to each of the 17 SDGs in turn, before ‘averaging’ the results into a single metric.A system of equal weights is deliberately employed to reflect international commitments ‘to treat each SDG equally and as an integrated and indivisible set of goals’ (Sachs et al., 2018, p. 41).

48. Although the SDG-I is not intended to replace the global dashboard of indicators for monitoring the SDGs (Sachs et al., 2017, p. 32), it does have huge potential (like other well-known composite indicators )– as for other renowned composite indicators, such as the UNDP Human Development Index, the OECD Better Life Index and the WEF Global Competitiveness Index, among many others – for identifying priority areas for action, tracking overall progress, and making international comparisons.

49. Human Development Index(HDI)  The Human Development Index (HDI) is a statistic composite index of life expectancy, education, and per capita income indicators, which are used to rank countries into four tiers of human development. A country scores a higher HDI when the lifespan is higher, the education level is higher, and the gross national income GNI (PPP) per capita is higher. It was developed  by Pakistani economist Mahbub ul Haq

50. Human Development Index(HDI) The HDI consists of three aspects of development: 1) Levels of wealth within the country as measures by GDP per capita and adjusted in Purchasing Power Parity (PPP). I.e., taking into account what a person is actually able to buy with a given income. 2) Education – measured by the percentage of the population in education at a particular age (Primary, secondary and tertiary) and literacy levels (educational attainment). 3) Health – life expectancy at birth. The HDI is expressed as a value between 0 and 1. The closer to 1 the score is, the higher level of human development.