Democracy Surveys as a Tool in Impact Evaluations

1. Democracy Surveys as a Tool in Impact Evaluations Promise and Challenges Dr. Margaret Sarles Margaretsarles@gmail.com US Agency for International Development Cairo, March 31, 2009

2. Presentation objectives: • Review the challenges of measuring impact of programs in democratic development • Prospects and cautions of using national democracy surveys as an impact evaluation tool • Dialogue: How can we make them more useful?

3. Why is this an important topic? • A growing consensus that democracy is a component of development, either as a core objective or to further economic development • Foreign policy priority for many countries: human rights, ROL, free and fair elections, democratic states, democratic cultures: policy imperatives and concern with trends • High investment levels but little scrutiny of effectiveness • Evaluations of democracy investments provide the evidence of what works, what doesn’t, under what circumstances: USAID commitment.

4. Measuring Impact on democratic change is difficult • The general concepts are contested and vague. What is success? At what level? (overall, components, programs) • Extremely diverse programs are grouped as “democracy” • community policing, strengthening civil society, elections, parties, justice systems, human rights, transparency, anti-corruption, decentralization • anything that could be said to be part of the political system or that contributes to democratic change (in any program) • “Politicized” definitions by stakeholders • *Practice outruns theory and academic analysis on processes, sequencing and causal links of democratic change variables (relevance)*

5. But the time is right to begin impact evaluations in democratic development • Volume of academic work is exploding. Case studies, quantitative work  hypotheses • Renewed donor emphasis on evaluation and evidence (including this conference) • USAID: (a) 20+ years’ experience  testable hypotheses, and (b) National Academy of Sciences recommendations and other studies • In addition to burgeoning hypotheses, more and better data are available, particularly democracy surveys.

6. Regional surveys are expanding • World Values surveys: 1981; 4 waves; 80+ countries; broad sociocultural scope • Afrobarometer: 1993; 4 rounds; 18 countries • Arab Barometer: 2005 “to produce scientifically reliable data on the politically relevant attributes of ordinary citizens….to contribute to political reform….” 1 round (2007), 5 countries. • Asian Barometer: 2001; 13 E. Asian and 5 S. Asian countries; 2 rounds • Latinobarometro: 1995, 18 countries, annual, views on economics, trade, politics • AmericasBarometer; 1992 (1970s) 23 countries; up to 10 rounds • Plus many independent country-level surveys

7. USAID has been the dominant donor supporting AmericasBarometer first as a policy tool (NOT evaluation) • Began with policy: The Peace Accords in Guatemala: The DIMS (now 6 rounds): what do citizens think about and expect of democracy? • Diagnostic tool for governments (legislatures, executive) and donors • Mobilizing tool/data for reform • Increased interest in democracy reform in society (media, NGOs; especially comparative) • “Voice” of the people in imperfect democracies • Reports are geared towards a policy audience • This becomes a winning scenario for increasing stakeholders and keeping support

8. But they did also provide excellent evaluation measures-- due to methodology - Time series data • Transparency and high standards (sample frame, pre-tests, etc.) • Inclusiveness (6 languages in Guatemala) • Allows multivariate analysis, and easy use for both qualitative and quantitative analysis • Learning from other surveys: concepts, definitions • Training of USAID democracy officers

9. And are good evaluation tools because of substance • Survey questions include (1) behavior (e.g.,voting, corruption, participation; (2) political culture (e.g.,system support); (3) knowledge; (4) attitudes (e.g.,trust); (5) personal attributes • USAID Democracy officers participated in questionnaire development, for relevance (now true for other donors)-- “Black,” “gray,” and “white” questions (ex:DLG). • Result: Used in Bolivia in over half the performance indicators of program success (ex.)

10. Improvements now make surveys useful for impact evaluations • Introduction of over-sampling in geographically based programs: 2004 • Allowed study of counterfactual for the first time • Richer comparisons (against national and other country data) • Therefore had before/during/after, and an approximation of with/without • Evidence that randomization may not decrease chances of project success (rarely implemented; now being investigated )

11. However, they require adaptation to be used in impact evaluation • Biggest requirement, and least understood, is over-sampling • Randomization is widely resisted • Questions (and indices) usually need to be added to be relevant to a particular intervention (often difficult) • Counterfactual methodologies need to be (1) developed and (2) costed out • Long-term commitment should be as guaranteed as possible, understanding that

12. Examples of Impact Evaluations Using National Surveys • Bolivia Democratic Development and Citizen Participation Project (1996 - 2003) • Over-sampled treated municipalities; compared to non-treated municipalities, to national sample, and to other countries. • Questions focused on participation and satisfaction • Kenya Civic Education Project • Innovative use of “non-treated” population through matching • Used surveys to determine beliefs and levels/kinds of participation Both evaluations used multiple methods, even though dominated by survey results

13. Nonetheless, there are significant challenges to using survey research in impact evaluations(1) Conceptual: Linking questions to (1) theory; (2) processes of democratic change Methodological: ensuring basic standards are met for design and implementation. Need for transparency Post-evaluation analysis is possible but late (Exs: Paraguay; LAPOP) New methods help get it right in the first place (PDAs, GPS, institution-building). national and trans-national design and methods

14. What are the challenges of using survey research in impact evaluations? (2) • Administrative: funding over project life (Guatemala); complex planning and budgeting for multi-country surveys; developing stakeholders in each country; • Maximizing use of them to justify costs, while recognizing surveys are usually not sufficient for measuring impact on democratic change

15. What should be done to improve surveys for impact evaluations? • Learn from the health model: periodic, institutionalized, health surveys with committed funding, widespread acceptance • Support research on democracy indicators derived from survey questions and indices (gap analysis, basic research) • Educate developmentalists on over-sampling, and other appropriate survey techniques (randomization, matching, etc.) • Institutionalize funding/administration of surveys, against the present dispersed, decentralized, often competing regional surveys

16. References • AmericasBarometer.com (All country and regional reports and data from Latin American surveys undertaken by Mitchell Seligson, Vanderbilt University) • Bollen, Ken et. Al. “Assessing International Evaluations: An Example from USAID’s Democracy and Governance Programs,” American Journal of Evaluation, 26: 189-203, 2005 • Chemonics International Inc. “Bolivia Democratic Development and Citizen Participation Final Report, 196-2003,” November 2003 • Finkel, Steven E. “The Impact of the Kenya National Civic Education Programme on Democratic Atttitudes, Knowledge, Values, and Behavior. “ Report prepared for US Agency for International Development, Nairobi, Kenya (AEP-I-00-00-00018, Task Order No. 806). December 30, 2003. • _________. “Kenya National Civic Education Impact Study: Report on Pre-Test, Prepared for US Agency for International Development, “Nairobi (Management Systems International and U. of Virginia) August 13, 2002. • National Research Council of the National Academies, Improving Democracy Assistance: Building Knowledge Through Evaluations and Research. National Academies Press, Washington, DC 2008 (available download on www.nap.edu) • Sarles, Margaret J. “Evaluating the Impact and Effectiveness of USAID’s Democracy and Governance Programmes,” in Peter Burnell, Evaluating Democracy Support: Methods and Experiences (IDEA and SIDA), 2007.