Designing Household Surveys

1. Designing Household Surveys Ec798 MA Global Development Capstone Course Lecture 3 DilipMookherjee

2. Outline • Motivation: Why do Household Surveys? • Develop check-list of steps in planning/thinking about survey design • Sample design issues (draw upon Deaton’s book on Household Surveys, Chapter 1) My own views and talking points: the kinds of issues that matter in practice but are not in survey design textbooks

3. Why Do Household Surveys? • Everybody in the development area (governments, aid agencies, academics) need to assess the impact of development programs and policies, to get a sense of ``what works’’ and what doesn’t • Strangely, this is something new among most development practitioners • Even the World Bank has set up program appraisal and impact evaluation as part of its routine operations only recently, so for most part their programs have never incorporated lessons from past experience

4. Household surveys of course have been around for far longer, going back to studies of living standards among the poor in the UK in the 1920s, and the Indian sample surveys in the 1940s: the World Bank has LSMS’s in many LDCs since the mid-80s • Purpose of those surveys was to get facts, on what per capita income, inequality, poverty really was in any given country at any given time • But they were rarely used as tools for evaluating development programs

5. Why are HH Surveys Important in Program Evaluation? • What are the criteria by which we should evaluate any given policy or program? • Can look at effects on some macro variables such as wage rates, employment levels, exports, or at a more micro-level at effects on communities or enterprises receiving assistance etc. (eg coop profits, roads and schools built in a community etc.)

6. Problems with all of these: don’t have any way to evaluate the impact directly on what we all care about ultimately: household living standards • Based on an implicit judgment that increased exports, profits, infrastructure etc of the region/enterprise/community do benefit households in some significant way • Data on exports/profits/infrastructure obtained from concerned communities/regional governments may be somewhat cooked to show results, so they need independent corroboration • And the distribution of benefits within the community or enterprise is also key to understanding impact on poverty reduction

7. Typically most enlightened aid agencies or governments using some kind of aggregative information on collective outcomes will concede that this is rather coarse, but that designing household surveys is difficult, expensive and time-consuming • Many agencies/policy-makers/media outlets do not even see the need for policy evaluation, or are happy to rely on personal experience/anecdotes/case studies/political reactions (eg Indian govt officials involved in implementing decentralization program) • Others don’t see the value of statistical information (Schelling’s distinction between statistical and personal lives)

8. Survey of What? • Household surveys usually involve a sampling frame to ensure that responses are representative of a larger population, and administer detailed questionnaires to those in the sample concerning their living standards and other variables that pertain to the program being evaluated • Attempt to obtain objective, factual information that is subject to minimal psychological, perceptual biases

9. Whats Wrong with Perception Surveys? • One simple way may be to ask households whether they think the program has been successful/good etc without going into details • Framing effects and inconsistencies documented by psychologists: these are magnified when perceptions are elicited • Problems of comparability across individuals/communities/countries: the importance of implicit benchmarks or standards that vary widely

10. Problems with Perception Surveys, contd. • Responses subject to mood swings, which magnify measurement error • Problems of credibility of the response: sometimes people pass judgment without thinking that much, sometimes they pronounce judgment to `make a statement’ even if they don’t sincerely believe what they say • Often they are not careful to consider counter-factuals, or to unbundle programs from others they `associate’ with it (eg my privatization project in LA) • Sometimes its hard for them to assess policy impact without a detailed listing and quantification of channels of impact (psychological aggregation problems, eg Robyn Dawes’s evidence)

11. Level of Aggregation: Who Should be Surveyed? • Household surveys also represent a way to collect impact assessment data at a more dis-aggregated level than community or enterprise level information • Provides assessment of distributive impacts, besides incorporate behavioral responses by households (eg effects on work, savings, investment etc.) which is key to understanding of channels of impact, sources of possible bias, and ultimate effects on well-being • Is this (i.e., household level) disaggregated enough?

12. Arguments for Surveying Individuals • Nowadays the argument is building for surveying individuals rather than households because of: • Interest in gender empowerment effects, effects on children • Households are getting fragmented, subdivided etc. and these processes of household division themselves need to be studied • Extensive increase in migration However this is even more difficult and expensive than surveying households (you have to track down and survey individuals)

13. Scope of Survey • Decide first what the purpose or scope of the survey is: be specific about (a) the programs to be evaluated and (b) the range of effects to be measured • Be sensitive to the specific context or area where it is to be carried out • Carry out exploratory visits, read background literature, talk to people on the ground, in settling on all this • Estimate costs, and compare with budget available (money, time, manpower) • Be brutally realistic about what is feasible both logistically as well as intellectually

14. Scoping, contd. • Look at datasets already available, and use them as departure point (e.g. last LSMS in the country) • Build on those questionnaires, but then add the questions you need to appraise the specific programs and program effects, while deleting those you do not need

15. Conceptual Framework: The Role of Theory • More often than not, survey design is not much influenced by any theory • I believe theory is important at all stages: it is necessary to frame the main hypotheses, select the key variables that need to be measured, think about household behavioral responses and resulting market/community/political effects that represent and influence channels of impact,anticipate key analytical and econometric problems you will face in making inferences from the data

16. Often its difficult to theorize a lot in advance of the actual survey or study, but at least you should try to state the key hypotheses or questions precisely, the variables that need to be measured, the regression specifications to be used, and the inferences that can be drawn from the exercise • This is part of the overall scoping exercise, an assessment of what you can feasibly expect to achieve, which should guide your survey design • Back-and-forth between theory and evidence, an endlessly iterative procedure

17. Various Theory-Data Interfaces/Methodologies • Most ambitious and audacious: structural modeling and estimation (complete model of motivation of individual agents, their behavior, aggregation/equilibrium at market/community/regional level, followed by welfare analysis and policy simulations eg CGE models) • Intermediate: reduced form modeling (relate observable impacts to policy/program treatments, household and community characteristics, time dummies etc, without pinpointing exact causal pathways) • Low: descriptive/factual (report facts and avoid any inferences about the actual impact or how it happened)

18. Survey Frames and Coverage • Decide on relevant population, sample size, basis of drawing samples • Context matters a lot (eg urban versus rural) • Power calculations (use prior information on means and variances to calculate the sample size needed to achieve a given degree of power) • Cost also matters (eg optimal design trades off precision versus cost: if rural hh’s are dispersed and expensive to get to, then you want fewer of them)

19. Strata and Clusters • Often stratify the population into areas or groups and draw a stratified random sample via a two step procedure (eg sample villages, then households) • Logistically and financially advisable (eg simple random sample of rural households would be very difficult and expensive) • Stratification also ensures representation of particular groups (eg if you want to guarantee some landless and some big landlords in the sample)

20. Missing Groups/Responses • In practice, cannot implement perfect representation of all groups (Deaton’s examples) and ways of correcting for these by re-weighting the sample • More important, be aware of biases that may creep in as a result of missing groups, some of which can be pretty subtle but profound (eg. the `dog that did not bark in the night’) • See if there are systematic coverage/non-response biases (eg regression of coverage on relevant characteristics)

21. Panel versus Cross-Section? • A key question concerns timing of coverage: should you carry out a one-time survey? • This is easiest and cheapest • But then you are not able to see the impact over time: like viewing a single snapshot rather than a film • It is difficult to assess impact from a one-time survey: have to rely on comparison of treatment and control areas, which typically raise thorny selection bias problems (e.g., most govt programs or NGOs do not randomly select treatment areas or households, but target them purposively)

22. If you carry out a study at two or more points of time, you can see `before-after’ changes, or impacts as they evolve over time • Repeated cross-sections or re-survey of the same set of households (a panel)? • Depends on whether the object of analysis is the community or the individual household • Typically it is the individual household, so a panel is highly desirable

23. Other Advantages of Panel Data • Can view impact on individual households over time (so can answer questions such as `how many people escaped poverty and how’?) • Reduce problems of inference substantially by controlling for unobserved household-level heterogeneity (farm-size productivity example, nutrition-education example) • Data that has been most instrumental in furthering academic research (PSID, ICRISAT, Progresa etc.) • Structure of Progresa intervention/data: treatment and control villages, then create a household panel within each

24. Problems with Panel Data • Can sometimes increase measurement error by focusing on temporal changes • Logistically and financially difficult • Problems of attrition: people move around a lot! • Intermediate solutions: • Revolving Panels (eg Mexican employment survey) • Recall data in one-time surveys (but then worry about recall errors)

25. Variables to be Measured • How to assess living standards or economic well-being: income, consumption, assets? • Income: problem of separating permanent from transitory income, of seasonality, and of measuring income • Consumption: again hard to do in practice (eg recall period?), but sometimes there is no other way (eg nutrition, health or gender empowerment programs) • Assets: lower measurement error but difficult to translate into income or consumption or intra-household implications

26. LSMS • Typical LSMS measures all of these: • Household demographics, housing, access to facilities, migration, • Consumption, education, health, anthropometrics, marriage • Employment, farming and livestock, non-farm enterprises • Credit and savings • Remittances and transfers Simple LSMS questionnaire runs into 80 pages

27. Practical Considerations • In my experience 80 pages are too many: people’s attention span and willingness to answer questions is limited to 3 hrs per sitting, in which you can administer at most a 30 page survey • Phrasing of questions very important (have to make sure the meaning is accurately conveyed, have to use local translators and local lingo): pilot the questionnaire and watch the subjects as they answer questions • Make questions as precise as possible, develop codes for qualitative answers

28. Steps • Listing of all relevant villages: random selection of villages with appropriate stratification • Listing of all households in selected villages, based on a 10-minute door-to-door survey of attributes you will stratify on (eg land owned, education, gender, occupation etc.) • Select stratified random sample of households in each village • Pilot the questionnaire, two or three times successively • Train surveyors

29. Steps, contd. • Manually check entered answers on an ongoing basis, get errors corrected ASAP • Computerization of data entry: select entry personnel, supervise their work with self-checking algorithms • Convert data to STATA or SAS or SPSS, then create relevant variables, create basic descriptive stats, tables and identify data errors: go back to data entry people and if necessary paper returns and original surveyors, and clean data • Now you are ready to start work!