Measuring food insecurity in the SDG era

1. Measuringfood insecurity in the SDG era Carlo Cafiero, PhD (former professor of Agricultural Economics and Policy, teaching statistics at the graduate level, currently senior statistician at FAO)

2. Outline • Food insecurity in the context of development agendas. From the MDGs to the SDGs • Measurement in the realm of social sciences • Validity and reliability of measures • The Rasch model • A critical review of existing food insecurity measures • Assessing the adequacy of dietary energy consumption • Measuring the severity of food insecurity through the lenses of people experiences

3. From the MDGs to the SDGs

4. From MDG-1 to SDG-2: much more than continuation of an advocacy campaign • MDG-1: Eradicate extreme poverty and hunger • Target 1.C: To halve the proportion of individuals suffering from hunger in the period between 1990 and 2015 • Indicator 1.8 Prevalence of underweight children under-five years of age • Indicator 1.9 Proportion of population below minimum level of dietary energy consumption • SDG-2: End hunger, achieve food security and improved nutrition and promote sustainable agriculture • Target 2.1: By 2030, end hunger and ensure access by all people, in particular the poor and people in vulnerable situations, including infants, to safe, nutritious and sufficient food all year round • Indicator 2.1.1 Prevalence of undernourishment • Indicator 2.1.2 Prevalence of moderate or severe food insecurity in the population, based on the Food Insecurity Experience Scale (FIES)

5. The SDG monitoring framework • Broader, more ambitious, and potentially more effective • 242 indicators, many targets aim at “zero” or “100%”, leaving no one behind, with national authorities in the driving seat But… • More demanding in terms of methods, standards and tools • Many new areas of interest, that are still not part of national statistical systems • Comparability of the indicators across countries is essential, to ensure meaningful aggregation • Indicators must be timely, relevant, scalable and reliable

6. Measuring food insecurity in a relevant, timely, reliable, cross-country comparable way • Existing indicators for MDGs did not fulfil the need created by the new demands • Malnutrition: • Children underweight largely insufficient as it confounds acute and chronic malnutrition, an • New indicators: stunting, wasting+overweight in children, anaemia in WRA, low-birthweight, exclusive breastfeeding, adolescent and adult overweight • Food access: • PoU: only national level, 2-3 years delay, insufficiently precise to capture very low levels (< 5%) • FCS, HDDS: lacking a basis for ensuring cross-country comparability • CSI, rCSI: measures of resilience to shock/intensity of shocks • HFIAS/HHS: lack cross-country comparability; relevant for acute f.i. • New indicator: FIES-based prevalence of food insecurity

7. Challenges • Statistical capacity • Improving food consumption data collection • Integrating the FIES module in national surveys • Analytic capacity • Deriving correct measures of the dietary energy consumption inadequacy • Calibrating measures of food insecurity against the FIES global reference scale • Recognizing the differences between the “old” and the “new” indicators • Broadening the scope • Food insecurity at moderate levels has significant implications for various forms of malnutrition, relevant for emerging and medium-high income economies too.

8. Measurement in the realm of social sciences

9. Measurement principles • Assigning numbers to objects to reflect a specific attribute (measurand) • A measurement system consists of a measurement tool and a protocol for application • Validity: measures reflect the magnitude of the attribute being measured. Ideally measures changes if and only if there is a change in the attribute being measured • Reliability: residual uncertainty around the measures is due to accidental factors • Precision: small errors • Accuracy: no bias

10. Types of measures • Nominal: binary variables that indicate belonging of the object to a certain class. No mathematical operation is meanigful • Ordinal: integer (ordinal) numbers reflecting the rank of the objects, based on the magnitude of the attribute. “Greater than”, “equal to” and “less than” are the admitted operations • Interval: numbers on a continuous scale reflect the relative magnitude of the attributes. Differences and sums are meaningful, but there is no inherent meaning in the unit of measure. No natural origin or metric. • Ratio: The scale has a natural or “absolute” origin. All operations (+, -, :, x) can be made with the measure. Ratios are meaningful.

11. Is measurement legitimate/possible in the social sciences? • Measurands in social sciences are not physical objects, but rather abstract conceptual constructs • Measurement is legitimate to the extent that object can be classified, ordered, compared with each other, and/or against a standard of reference Claim: “Measuring” is a very abused word. Too often, numbers are associated to objects in the social sciences, and treated as if they were measures when in reality, they are not. Claim: Measurement can only be conceived as inherently probabilistic. There will always be residual uncertainty around the measures

12. Fundamental measurement in the social science • Validity amounts to defining a convincing model that links the observables (the “evidence”, or “data”) to the attribute being measured. • Proponents must explain how the things wee can see are linked to the things we would like to measure • Reliability amounts to evaluating the extent, magnitude and direction of discrepancies between the measure and the “true” magnitude of the measurand. • The model can only be framed probabilistically

13. The data, , is the “response” of the i-th respondent to the j-th “item”. The probability that the subject whose attribute’s position on a continuous scale is , might respond to an item positioned at on the same scale is a (logistic) function of the difference The Rasch model (G. Rasch, 1960)

14. \ \ \ Item 1 Item 2 Item 3

15. The Rasch model • The probability to affirm an item is increasing in the distance between the item and the respondent • Example: • The more “competent” is a student, the morel likely it is that she will answer correctly any item • “Easier” items will be answered correctly more often than “difficult” ones • Measures are defined/produced on an interval scale • As the probability depends only on the difference between measures, the model is defined up to an arbitrary constant • Maximum likelihood principles can be applied to estimate the values of and , given a set of data .

16. “Fundamental” measures in the social sciences • Rasch principles establish the possibility to have fundamental (as opposed to “derived”) measurement in social sciences (see the discussion in Bond and Fox, 2015) • The Rasch model imposes a strong structure on the relation between observable data and the measure. • This is like the “rigidity” of a stick that is used to make measures of “linear length”. (No one would want to use an elastic band as a ruler) • Equal discrimination and conditional independence of the responses to the items ensures measure invariance: the way in which the measurement system works does not depend on the object being measured.

17. A critical review of food insecurity measures

18. Discussing validity: the need to uniquely define the attribute being measured • Food security as a ‘complex’ phenomenon • Availability, Access, Utilization, Stability … useful conceptual framework, not conducive to an operational definition of a measurable attribute • Whose food security? (The object) • Global? A country? A population group? An individual? • What to measure? (The measurand) • Adequacy of food consumption? In quantity? Quality? Continuity? Stability? Security? • Nutritional consequences of food consumption? Psychological implications of the inability to access food? Social status? • All of the above?

19. Validity as separate from reliability • Most discussions on food security measurement/assessment have been dominated by a view that is profoundly qualitative. • With few exceptions, there has never been a proper statistical model to link the data to the measure Attempts at quantification have led to either denying the possibility to measure (“there is no ‘gold standard’”) or arbitrarily defining numeric “scores” or “indexes” to be “validated” ex post • Ex-post validation has been attempted by combining (confusing?) issues of validity (e.g., definition of the attribute being measured and relevance of the data uses) with those of reliability (that is, whether the measures obtained were good enough for the purpose at hand)

20. Three main concepts • Food security indicators have been created by focusing on three concepts: food consumption (look at what people eat), anthropometry (how people look) and food security experiences (how people behave) • Anthropometry provides data to infer on the nutritional status (not the food insecurity) of people • Food consumption is also one of the consequences of the “food security” status • Food insecurity experiences are the only way to directly measure food insecurity

21. Analyzing food consumption data

22. Motivation • Over the last two decades, household surveys have been increasingly used as a source of data to inform food consumption inadequacy assessments • An IFPRI’s manual (Smith and Subandoro 2007) has become a very popular reference for researchers and organizations who analyze food consumption data from household expenditure surveys • The methods described therein, however, lead to systematic overestimation of the prevalence of food energy deficiency in the population (i.e., the percentage of people who are food energy deficient) • The manual is still widely used, despite the fact that my predecessors at FAO (and myself) have repeatedly tried to expose its conceptual flaws • I shall try to illustrate the nature of the errors and present ways to correct them, including by making reference to FAO traditional PoU method

23. “[T]he percentage of households in a population group who do not consume sufficient dietary energy […] is measured by determining whether a household acquires sufficient food over the reference period to meet the dietary energy requirements of all of its members. If the estimated total energy in the food that the household acquires daily is lower than the sum of its members’ daily requirements, the household is classified as food energy deficient. The requirements employed are those for basal metabolic function (a state of complete rest) and light activity, such as sitting and standing. When the percentage of people, as opposed to households, is measured, each person is assigned the energy deficiency status of her or his household.” (Smith and Subandoro 2007, page 5) “As discussed in Chapter 2, use of the energy requirements for light activity is recommended” (ibid., page 64)

24. A normative error: Energy requirements should be referred to normally active life. “[T]he percentage of households in a population group who do not consume sufficient dietary energy […] is measured by determining whether a household acquires sufficient food over the reference period to meet the dietary energy requirements of all of its members. If the estimated total energy in the food that the household acquires daily is lower than the sum of its members’ daily requirements, the household is classified as food energy deficient. The requirements employed are those for basal metabolic function (a state of complete rest) and light activity, such as sitting and standing. When the percentage of people, as opposed to households, is measured, each person is assigned the energy deficiency status of her or his household.” (Smith and Subandoro 2007, page 5) “As discussed in Chapter 2, use of the energy requirements for light activity is recommended” (ibid., page 64)

25. A conceptual error: • Thresholds to be used to determine inadequacy are not the recommended intake levels. • Recommended levels for a given sex-age group refer to the average body mass of healthy individuals (BMI = 21.5) • Using the average as a threshold, would yield a prevalence of inadequacy of about 50% even in a perfectly nourished population • When no info exists on the ideal BMI of an individual, any level of dietary energy intake that is compatible with requirements for BMIs in the range 14.5-24.5 must be considered adequate.

26. “The variability in food acquisition data is far higher than that in food consumption data. Estimates of per capita energy availability from food acquisition data can legitimately range from 0 to 12,000 kilocalories per capita or more, far outside the range of appropriate energy consumption for human beings. For this reason, estimates of food energy deficiency can theoretically be different depending on which data source is used. In particular, if the population energy requirement is below the mode of the energy availability distribution, estimates of the prevalence of food energy deficiency are biased upward and vice versa. As mentioned earlier, current evidence suggests that this is not a major issue in most cases. The increased variability of HES food acquisition data would have to be quite large to lead to a meaningful difference in estimates of food energy deficiency. Nevertheless, the existence and magnitude of this bias need to be investigated further.” Likely a consequence of the combination of the two errors

27. Then, what is the proper way to determine energy intake adequacy? • Normative energy requirement for Basic Metabolic Rate are established per unit of body mass • This means that people of the same sex, age and height may have different energy requirements, depending on their “natural” bodily conformation • Defining energy requirements for a group of people requires that relevant characteristics of the group member are specified. In any case, only a range of values can be established, with a minimum (MDER) and a maximum (XDER) • In empirical assessments, when dietary energy intake must thus be considered: • insufficient, if it is below the MDER • excessive, if it is above the XDER

28. If a group is only defined in terms of sex and age, the rangeof requirements can be quite wide • BMI alone can induce differences between the MDER and XDER of about 25% of the average • For example, for people with moderately active lives (PAL = 1.75) • ADER = 2350 • MDER ≈ 2350/21.5*18.5 = 2020 • XDER ≈ 2350/21.5*24.5 = 2680 • XDER – MDER = 660 • Physical activity level can induce additional variability as “normally active” PALs can vary from 1.6 to 2.2 (a range of more than 30% of the mean) • Combining both PAL and BMI: • MDER = 2350/(21.5*1.75) * (18.5*1.6) ≈ 1850 • XDER = 2350/(21.5*1.75) * (24.5*2.2) ≈ 3360

29. Difference between recommended level and inadequacy threshold • The recommended daily dietary energy intake level must equal the average or the range (ADER), to balance the risks of over and under consumption • The 2100 Kcal/day value is neither a normative recommended level in normal conditions, nor the proper threshold to establish inadequacy • It was determined as the mean per capita daily energy requirement for emergency situations in a 1995 report by the Committee on International Nutrition, Food and Nutrition Board, Board on International Health (CIN, 1995) • It was then chosen as the dietary energy content of the typical food ratio for distribution, which “would cover the energy needs of a “typical” population, assuming standard population distribution, body size, ambient temperature, pre-emergency nutritional status and a light physical activity level of 55 percent above BMR for males and 56 percent for females.” (UNHCR/WFP, 1997) • It corresponds to the average requirement for an average individual with low physical activity: 2350/1.75*1.6 ≈ 2100

31. Population without undernourishment: Variability in food intake simply reflects variability in requirements Overestimation for using too high of a threshold Population with undernourishment: Variability, at the low end of the distribution, reflects the extent of undernourishment. Prevalence of Undernourishment MDER XDER 2100 ADER

32. Distribution with measurement errors Overestimationduetoexcessvariability True distribution PoU MDER

33. The FAO method • If it were possible to reliably observe/measure both dietary energy intake, , and requirement, on a sample of individuals that is representative of the population, estimating the prevalence of undernourishment would be a simple matter of counting: PoUFAO = [1]

34. The FAO method • When individual consumption cannot be matched to individual requirements, it has been proposed that the method could be applied anyhow, by substituting the unobserved requirement, , with the average requirement for the specific category group PoUALT = [2] • But this is wrong. Imagine a hypothetical population where everyone eats according to their requirements. By construction, true PoU is 0%, but, as there will be variation in the individual intakes, reflecting their requirements, POUALT50%

35. The FAO method • The challenge is how to ‘disentangle’ the variability in observed dietary intakes that is due to inequality in accessing food, form the component that simply reflects the diversity in the population • One solution is to try and estimate how much the differences in energy requirements among the members of the population or group might contribute to the variability in intakes, and take that into consideration when drawing inference from observed food consumption

36. The FAO method • FAO method implements this solution by defining a probability distribution model for the habitual food consumption of the average individual in the group, and estimating the probability associated with intake levels that are below requirements • For the average individual, there will be no single value of requirements, ri but rather a whole range {MDER, XDER} of values that are compatible with good health and normal physical activity in the population • The probability that intakes for the representative average individual fall below MDER is used as an estimate of the PoU in the population

37. f(x) PoU MDER

38. The FAO method In interpreting the FAO method, several things should be kept in mind: • The function is not the empirical distribution of daily habitual consumption values in a population; it is a statistical model of the population • (i.e., the fact that the estimate is taken to be the area to the left of the MDER does not implies that all who eat an amount less than the MDER are undernourished, and those who eat more are not) • The method is not a “headcount” approach, that is, it only applies to the group as a whole, and cannot be used to identify who, within the group, is undernourished • The reliability of the estimate crucially depends on the quality and completeness of the data used to estimate its parameters

39. The FAO method • Choice of the distribution • It should represent the probability distribution associated with habitual daily dietary energy consumption of the average, representative individual in the population • Could be indifferently expressed on a per caput or on a per adult equivalent basis (it is only a matter of scaling) • It can only be positively valued • It should have (strictly positive) natural upper and lower limits and be positively, but not excessively skewed (as opposed to, say, income, there is a natural limit to how much calories a human body can consume) • The log normal model was adopted, in 1996, for analytic convenience and goodness of fit, based on the few existing food intake surveys

40. The FAO method • Estimation of the parameters • Mean consumption estimated from Food Balance Sheets • Still the preferred option • Issues of coverage (i.e. non commercial production, accounting for losses etc.), precision (unreported trade, stocks). • CV of food consumption • Because of problems in properly dealing with survey data, the CVs of food consumption had not been revised for most countries since 1999 • A revision of the CV values for India was conducted in 2009 • A complete review and revision of the values for most countries in 2012 • MDER • Estimated from external data on the population pyramid (Population Division of UNSD) and on the population height

41. Towards a household survey based, headcount approach • When all information on dietary energy consumption and on the sex-age structure of the population is taken from a survey, the PoU method is equivalent to a method that classifies each household as dietary energy deficient if the per capita DEC in the household is below the household specific MDER, and then counts the proportion of people leaving in households thus classified • To the extent that (a) the survey is representative of the national population, and (b) food consumption data are highly reliable measures of the households habitual food consumption level, such a procedure would yield an estimate that is comparable to the PoU as estimated by FAO • The extant problem is how to treat measurement error and the discrepancy between occasional and habitual household level food consumption data

42. Towards a household survey based, headcount approach • Per capita dietary energy acquisition (), computed separately for each household as described in Smith and Subandoro (2017), can be modeled as the combination of the per capita level of habitual dietary energy consumption (), of a transitory component due to possible seasonality in food consumption (), and of measurement error () • In turn, can be modeled as a function of relevant household characteristics () • Using the predicted value (instead of to classify households

43. Experience-based food security scales (EBS)

44. The severity of the food insecurity condition as a latent trait • Through ethnographic research, Radimer and colleagues (Radimer et al 1990, 1992) established consistency of typical experiences associated with food insecurity • In 1995, using Rasch model, USDA proposed creating a measurement scale using 18 survey items that were included in the Current Population Survey, creating the HFSSM • It has been used officially in the US since 1997 • Other scales (HFIAS, ELCSA, EBIA, EMSA) have been derived as adaptations of the HFSSM • In 2014, FAO established the Food Insecurity Experience Scale (FIES) as a global reference standard, thus finally providing the possibility to

45. FIES genealogy U.S. Household Food Security Survey Module USA, 1995; Canada, 2004 Colombia Venezuela Household Food Insecurity Access Scale HFIAS EMSA Mexico, 2008 ELCSA Guatemala, 2011 EBIA Brazil, 2004 FIES A global reference standard HHS

46. Thanks Carlo.cafiero@fao.org