Facilitated by Oumoul Ba Tall and Jim Rugh Note: This PowerPoint presentation and the summary (condensed) chapter of the

1. Promoting a RealWorld and Holistic approach to Impact EvaluationDesigning Evaluations under Budget, Time, Data and Political ConstraintsWorkshop for staff of MFA of FinlandHelsinki, 27-28 September 2012 Facilitated by Oumoul Ba Tall and Jim Rugh Note: This PowerPoint presentation and the summary (condensed) chapter of the book are available at: www.RealWorldEvaluation.org

2. Introduction to Impact Evaluation(quoting NONIE Guidelines) • “In international development, impact evaluation is principally concerned with final results of interventions (programs, projects, policy measures, reforms) on the welfare of communities, households and individuals.” [p. 3] • “No single method is best for addressing the variety of questions and aspects that might be part of impact evaluations. However, depending on the specific questions or objectives of a given impact evaluation, some methods have a comparative advantage over others in analyzing a particular question or objective. Particular methods or perspectives complement each other in providing a more complete ‘picture’ of impact.” [p. xxii]

3. Introduction to Impact Evaluation(quoting NONIE Guidelines) • Three related problems that quantitative impact evaluation methods attempt to address are the following: • The establishment of a counterfactual: What would have happened in the absence of the intervention(s)? • The elimination of selection effects, leading to differences between the intervention (treatment) group and the control group. • A solution for the problem of unobservables: The omission of one or more unobserved variables, leading to biased estimates. [p. 23]

4. Introduction to Impact Evaluation(quoting NONIE Guidelines) • “The safest way to avoid selection effects is a randomized selection of the intervention and control groups before the experiment starts. In a well-designed and correctly implemented Randomized Controlled Trial (RCT) a simple comparison of average outcomes in the two groups can adequately resolve the attribution problem and yield accurate estimates of the impact of the intervention on a variable of interest. By design, the only difference between the two groups was the intervention.” [p.24]

5. OK, that’s enough of an initial introduction to the theory of Impact Evaluation. More reference is provided at the end of this presentation for further research on IE Let’s begin to consider some implications of trying to conduct Impact Evaluations in the RealWorld.

6. Applying Impact Evaluation to this Workshop All of you in this room have chosen to participate in this workshop. We’ll call the subject matter we cover during this workshop the intervention. We presume that you are all adequately qualified to take this course. So here is what we propose to do: We will issue a pre-test to determine how well you know this subject matter. Then we will randomly choose 50% of you whom we will identify as our control group. We will ask you to leave this room and do whatever else you want to do during the next two days. Then we will ask you to come back at 4:30 pm tomorrow to take the post-test, along with those who stayed and participated in the rest of this workshop (our intervention group). Thus we will have a counterfactual against which to measure any measurable ‘impact’ of what we taught the workshop

7. Applying Impact Evaluation to this Workshop OK, just kidding, to make a point!

8. Applying Impact Evaluation to this Workshop What are reasons why attempting to use a Randomized Control Trial to evaluate the impact of this workshop might not be a good idea? …

9. Workshop Objectives 1. The basics of the RealWorld Evaluation approach for addressing common issues and constraints faced by evaluators such as: when the evaluator is not called in until the project is nearly completed and there was no baseline nor comparison group; or where the evaluation must be conducted with inadequate budget and insufficient time; and where there are political pressures and expectations for how the evaluation should be conducted and what the conclusions should say;

10. Workshop Objectives • Defining what impact evaluation should be; • Identifying and assessing various design options that could be used in a particular evaluation setting; • Ways to reconstruct baseline data when the evaluation does not begin until the project is well advanced or completed; • How to account for what would have happened without the project’s interventions: alternative counterfactuals • The advantages of using mixed-methods designs

11. Workshop Objectives Note: In this workshop we focus on project-level impact evaluations. There are, of course, many other purposes, scopes, evaluands and types of evaluations. Some of these methods may apply to them, but our examples will be based on project impact evaluations, most of them in the context of developing countries.

12. Workshop agenda Day 1 Introduction [10 minutes] Brief overview of the RealWorld Evaluation (RWE) approach [30 minutes] Rapid review of evaluation designs, logic models, tools and techniques for RealWorld impact evaluations, with an emphasis on impact evaluation (IE); overview of range of evaluation models for IE compared to other types of evaluations [75 minutes] --- short break [20 minutes]--- Small group introductions and sharing of RWE types of constraints you have faced in your own practice, and how they were addressed. [45 minutes] What to do if there had been no baseline survey: reconstructing a baseline [30 minutes] --- lunch [60 minutes] --- Small group exercise Part I: read case studies and begin discussions [45 minutes] Small group exercise Part II: 'Clients’ and ‘Consultants’ re-negotiate the case study evaluation ToR [45 minutes] Feedback from exercise [20 minutes] Wrap-up discussion, evaluation of Day 1, plans for Day 2 [40 minutes]

13. Workshop agenda Day 2 Review and reflections on topics addressed in Day 1 [25 minutes] Quantitative, qualitative and mixed methods  [30 minutes] How to account for what would have happened without the project: alternative counterfactuals [30 minutes] --- short break [20 minutes]--- Considerations for more holistic approaches to impact evaluation [45 minutes] Check lists for assessing and addressing threats to validity [20 minutes] Plenary discussion: Practical realities of applying RWE approaches: challenges and strategies [30 minutes] --- lunch [60 minutes] --- Small-group work applying RealWorld Evaluation methodologies for the Norad Nepal evaluation case study [120 minutes] Next steps: how to apply what we’ve learned in the real worlds of our work [20 minutes] Workshop evaluation and wrap-up [20 minutes]

14. RealWorld EvaluationDesigning Evaluations under Budget, Time, Data and Political Constraints Session 2 OVERVIEW OF THE RWE APPROACH

15. RealWorld Evaluation Scenarios Scenario 1: Evaluator(s) not brought in until near end of project For political, technical or budget reasons: • There was no life-of-project evaluation plan • There was no baseline survey • Project implementers did not collect adequate data on project participants at the beginning nor during the life of the project • It is difficult to collect data on comparable control groups

16. RealWorld Evaluation Scenarios Scenario 2: The evaluation team is called in early in the life of the project But for budget, political or methodological reasons: • The ‘baseline’ was a needs assessment, not comparable to eventual evaluation • It was not possible to collect baseline data on a comparison group

17. Reality Check – Real-World Challenges to Evaluation • All too often, project designers do not think evaluatively – evaluation not designed until the end • There was no baseline – at least not one with data comparable to evaluation • There was/can be no control/comparison group. • Limited time and resources for evaluation • Clients have prior expectations for what they want evaluation findings to say • Many stakeholders do not understand evaluation; distrust the process; or even see it as a threat (dislike of being judged)

18. RealWorld Evaluation Quality Control Goals • Achieve maximum possible evaluation rigor within the limitations of a given context • Identify and control for methodological weaknesses in the evaluation design • Negotiate with clients trade-offs between desired rigor and available resources • Presentation of findings must acknowledge methodological weaknesses and how they affect generalization to broader populations

19. The Need for the RealWorld Evaluation Approach • As a result of these kinds of constraints, many of the basic principles of rigorous impact evaluation design (comparable pre-test -- post test design, control group, adequate instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology etc.) are often sacrificed.

20. The RealWorld Evaluation Approach An integrated approach to ensure acceptable standards of methodological rigor while operating under real-world budget, time, data and political constraints. See the RealWorld Evaluation book or at least condensed summary for more details

21. Bamberger Rugh Mabry RealWorld Evaluation RealWorld Evaluation Working Under Budget, Time, Data, and Political Constraints Michael Bamberger Jim Rugh Linda Mabry EDITION 2 EDITION This book addresses the challenges of conducting program evaluations in real-world contexts where evaluators and their clients face budget and time constraints and where critical data may be missing. The book is organized around a seven-step model developed by the authors, which has been tested and refined in workshops and in practice. Vignettes and case studies—representing evaluations from a variety of geographic regions and sectors—demonstrate adaptive possibilities for small projects with budgets of a few thousand dollars to large-scale, long-term evaluations of complex programs. The text incorporates quantitative, qualitative, and mixed-method designs and this Second Edition reflects important developments in the field over the last five years. New to the Second Edition: • Adds two new chapters on organizing and managing evaluations, including how to strengthen capacity and promote the institutionalization of evaluation systems • Includes a new chapter on the evaluation of complex development interventions, with a number of promising new approaches presented • Incorporates new material, including on ethical standards, debates over the “best” evaluation designs and how to assess their validity, and the importance of understanding settings • Expands the discussion of program theory, incorporating theory of change, contextual and process analysis, multi-level logic models, using competing theories, and trajectory analysis • Provides case studies of each of the 19 evaluation designs, showing how they have been applied in the field “This book represents a significant achievement. The authors have succeeded in creating a book that can be used in a wide variety of locations and by a large community of evaluation practitioners.” —Michael D. Niles, Missouri Western State University “This book is exceptional and unique in the way that it combines foundational knowledge from social sciences with theory and methods that are specific to evaluation.” —Gary Miron, Western Michigan University “The book represents a very good and timely contribution worth having on an evaluator’s shelf, especially if you work in the international development arena.” —Thomaz Chianca, independent evaluation consultant, Rio de Janeiro, Brazil 2 EDITION RealWorld Evaluation

22. The RealWorld Evaluation approach • Developed to help evaluation practitioners and clients • managers, funding agencies and external consultants • Still a work in progress (we continue to learn more through workshops like this) • Originally designed for developing countries, but equally applicable in industrialized nations

23. Special Evaluation Challenges in Developing Countries • Unavailability of needed secondary data • Scarce local evaluation resources • Limited budgets for evaluations • Institutional and political constraints • Lack of an evaluation culture (though evaluation associations are addressing this) • Many evaluations are designed by and for external funding agencies and seldom reflect local and national stakeholder priorities

24. Expectations for « rigorous » evaluations Despite these challenges, there is a growing demand for methodologically sound evaluations which assess the impacts, sustainability and replicability of development projects and programs. (We’ll be talking more about that later.)

25. Most RealWorld Evaluation tools are not new— but promote a holistic, integrated approach • Most of the RealWorld Evaluation data collection and analysis tools will be familiar to experienced evaluators. • What we emphasize is an integrated approach which combines a wide range of tools adapted to produce the best quality evaluation under RealWorld constraints.

26. What is Special About the RealWorld Evaluation Approach? • There is a series of steps, each with checklists for identifying constraints and determining how to address them • These steps are summarized on the following slide and then the more detailed flow-chart …

27. The Steps of the RealWorld Evaluation Approach Step 1: Planning and scoping the evaluation Step 2: Addressing budget constraints Step 3: Addressing time constraints Step 4: Addressing data constraints Step 5: Addressing political constraints Step 6:Assessing and addressing the strengths and weaknesses of the evaluation design Step 7: Helping clients use the evaluation

28. TheReal-World Evaluation Approach • Step 1: Planning and scoping the evaluation • . Defining client information needs and understanding the political context • . Defining the program theory model • . Identifying time, budget, data and political constraints to be addressed by the RWE • . Selecting the design that best addresses client needs within the RWE constraints Step 2 Addressing budget constraints A. Modify evaluation design B. Rationalize data needs C. Look for reliable secondary data D. Revise sample design E. Economical data collection methods Step 3 Addressing time constraints All Step 2 tools plus: F. Commissioning preparatory studies G. Hire more resource persons H. Revising format of project records to include critical data for impact analysis. I. Modern data collection and analysis technology • Step 4 • Addressing data constraints • . Reconstructing baseline data • . Recreating comparison groups • . Working with non-equivalent comparison groups • . Collecting data on sensitive topics or from difficult to reach groups • . Multiple methods Step 5 Addressing political influences A. Accommodating pressures from funding agencies or clients on evaluation design. B. Addressing stakeholder methodological preferences. C. Recognizing influence of professional research paradigms. Step 6 Assessing and addressing the strengths and weaknesses of the evaluation design An integrated checklist for multi-method designs A. Objectivity/confirmabilityB. Replicability/dependabilityC. Internal validity/credibility/authenticityD. External validity/transferability/fittingness Step 7 Helping clients use the evaluation A. Utilization B. Application C. OrientationD. Action 28

29. We will not have time in this workshop to cover all those steps We will focus on: • Scoping the evaluation • Evaluation designs • Logic models • Reconstructing baselines • Mixed methods • Alternative counterfactuals • Realistic, holistic impact evaluation • Challenges and Strategies

30. Planning and Scoping the Evaluation • Understanding client information needs • Defining the program theory model • Preliminary identification of constraints to be addressed by the RealWorld Evaluation

31. Understanding client information needs Typical questions clients want answered: • Is the project achieving its objectives? • Is it having desired impact? • Are all sectors of the target population benefiting? • Will the results be sustainable? • Which contextual factors determine the degree of success or failure?

32. Understanding client information needs A full understanding of client information needs can often reduce the types of information collected and the level of detail and rigor necessary. However, this understanding could also increase the amount of information required!

33. Still part of scoping: Other questions to answer as you customize an evaluation Terms of Reference (ToR): • Who asked for the evaluation? (Who are the key stakeholders)? • What are the key questions to be answered? • Will this be mostly a developmental, formative or summative evaluation? • Will there be a next phase, or other projects designed based on the findings of this evaluation? 33

34. Other questions to answer as you customize an evaluation ToR: • What decisions will be made in response to the findings of this evaluation? • What is the appropriate level of rigor? • What is the scope / scale of the evaluation / evaluand (thing to be evaluated)? • How much time will be needed / available? • What financial resources are needed / available? 34

35. Other questions to answer as you customize an evaluation ToR: • Should the evaluation rely mainly on quantitative or qualitative methods? • Should participatory methods be used? • Can / should there be a household survey? • Who should be interviewed? • Who should be involved in planning / implementing the evaluation? • What are the most appropriate forms of communicating the findings to different stakeholder audiences? 35

36. Evaluation (research) design? Resources available? Key questions? Time available? Evaluand (what to evaluate)? Skills available? Qualitative? Participatory? Quantitative? Extractive? Scope? Appropriate level of rigor? Evaluation FOR whom? Does this help, or just confuse things more? Who said evaluations (like life) would be easy?!! 36

37. Before we return to the RealWorld steps, let’s gain a perspective on levels of rigor, and what a life-of-project evaluation plan could look like 37

38. Different levels of rigor depends on source of evidence; level of confidence;use of information Objective, high precision – but requiring more time & expense Quick & cheap – but subjective, sloppy Level 5: A very thorough research project is undertaken to conduct in-depth analysis of situation; P= +/- 1% Book published! Level 4: Good sampling and data collection methods used to gather data that is representative of target population; P= +/- 5%Decision maker reads fullreport Level 3: A rapid survey is conducted on a convenient sample of participants; P= +/- 10%Decision maker reads 10-page summary of report Level 2: A fairly good mix of people are asked their perspectives about project; P= +/- 25%Decision maker reads at least executive summary of report Level 1: A few people are asked their perspectives about project; P= +/- 40%Decision made in a few minutes Level 0:Decision-maker’s impressions based on anecdotes and sound bytes heard during brief encounters (hallway gossip), mostly intuition; Level of confidence +/- 50%; Decision made in a few seconds 38

39. CONDUCTING AN EVALUATION IS LIKE LAYING A PIPELINE Randomized sample selection Reliability & validity of indicators Questionnaire quality Depth of analysis Reporting & utilization Quality of data collection QUALITY OF INFORMATION GENERATED BY AN EVALUATION DEPENDS UPON LEVEL OF RIGOR OF ALL COMPONENTS

40. Relevance & validity of indicators Randomized sample selection Questionnaire quality Depth of analysis Reporting & utilization Quality of data collection AMOUNT OF “FLOW” (QUALITY) OF INFORMATION IS LIMITED TO THE SMALLEST COMPONENT OF THE SURVEY “PIPELINE”

41. High rigor 4 3 2 Low rigor Time during project life cycle Determining appropriate levels of precision for events in a life-of-project evaluation plan Same level of rigor Final evaluation Baseline study Mid-term evaluation Special Study Needs assessment Annual self-evaluation 41

42. RealWorld EvaluationDesigning Evaluations under Budget, Time, Data and Political Constraints Session 3.a EVALUATION DESIGNS

43. So what should be included in a “rigorous impact evaluation”? • Direct cause-effect relationship between one output (or a very limited number of outputs) and an outcome that can be measured by the end of the research project?  Pretty clear attribution. … OR … • Changes in higher-level indicators of sustainable improvement in the quality of life of people, e.g. the MDGs (Millennium Development Goals)?  More significant but much more difficult to assess direct attribution.

44. So what should be included in a “rigorous impact evaluation”? OECD-DAC (2002: 24) defines impact as “the positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. These effects can be economic, sociocultural, institutional, environmental, technological or of other types”. Does it mention or imply direct attribution? Or point to the need for counterfactuals or Randomized Control Trials (RCTs)?

45. Some of the purposes for program evaluation • Formative: learning and improvement including early identification of possible problems • Knowledge generation: identify cause-effect correlations and generic principles about effectiveness. • Accountability: to demonstrate that resources are used efficiently to attain desired results • Summative judgment: to determine value and future of program • Developmental evaluation: adaptation in complex, emergent and dynamic conditions -- Michael Quinn Patton, Utilization-Focused Evaluation, 4th edition, pages 139-140

46. Determining appropriate (and feasible) evaluation design Based on the main purpose for conducting an evaluation, an understanding of client information needs, required level of rigor, and what is possible given the constraints, the evaluator and client need to determine what evaluation design is required and possible under the circumstances.

47. Some of the considerations pertaining to evaluation design 1: When evaluation events take place (baseline, midterm, endline) 2. Review different evaluation designs (experimental, quasi-experimental, other) 3. Levels of rigor 4. Qualitative & quantitative methods 5. A life-of-project evaluation design perspective.

48. scale of major impact indicator An introduction to various evaluation designs Illustrating the need for quasi-experimental longitudinal time series evaluation design Project participants Comparison group baseline end of project evaluation post project evaluation 48

49. OK, let’s stop the action to identify each of the major types of evaluation (research) design … … one at a time, beginning with the most rigorous design. 49

50. First of all: the key to the traditional symbols: • X = Intervention (treatment), I.e. what the project does in a community • O = Observation event (e.g. baseline, mid-term evaluation, end-of-project evaluation) • P (top row): Project participants • C (bottom row): Comparison (control) group Note: the 7 RWE evaluation designs are laid out on page 8of the Condensed Overview of the RealWorld Evaluation book 50