How to measure impact? • Assessing causality Impact of program = outcome 1 – outcome 2 In practice, we compare two groups, one of which benefited from the program, the other one did not Event 2 (Effects) Outcome 1 Event 1 e.g. Education program/ treatment Causes Event 2 occurs if and only if Event 1 occurred before Event 1 No Education Program Event 2 Outcome 2 Causes
Constructing the counterfactual • The counterfactual • what would have happened in the absence of the program? … • … for the people who benefitted from the program: we don’t observe it • thus, impact evaluation will have to mimic it • Counterfactual is constructed by selecting a group not affected by the program – this is the main challenge of impact evaluation • Methods differ by the way the counterfactual is constructed
Non-Experimental Methods • Differences between outcomes for treated and non-treated (in our ex: reached / not) are the sum of • Inherent differences: “SELECTION” BIAS • TREATMENT EFFECT: that we want to isolate from 1 • The essence of non-experimental methods is to find a way to redress the selection bias ex-post.
Non-Experimental Methods • Simple Difference • Multivariate (Multiple) Regression • “Difference in difference” (Multiple Regression with Panel Data) • Matching • Randomization/RCT’s (advanced topic covered separately)
Simple difference • Simple difference is a first measure – why is it not sufficient? • There may be differences between the two groups (age, location, gender, initial endowment, bargaining power) • So we may want to control for these differences
Multi-variate regression • Suppose we can observe these differences. • Age composition of the group, initial infrastructure in the school, level of education, … • We can include all these variables in a regression: Y = a.T + b.Age + c.Infr + d.Edu +… • A regression provides the linear combination of observable variables that best “mimics” the outcome • Each coefficient represents the effect of each variable • Will give us the effect of the treatment everything else being equal, or more exactly every other observable characteristics being equal
Multi-variate regression • Problems of the regression • You may want to include many many variables, to control for as many characteristics as possible • Problem of sample size (degrees of freedom) • More important: do you have measures of everything? • Bargaining power, Pro-activeness, Intrinsic motivation, hopelessness • There are unobservables
Panel data • Simple difference: before/after • Counterfactual = same group before the program • Can we trust this? Assumption = would have remained the same • Ex: Police project • Double difference • Control for the situation before the program • Ex: Group 1 = Treatment Group; Group 2 = Control Group • 2006: Group 1 : 30 Group 2 : 60 2009: Group 1 : 50(+66%) Group 2 : 90 (+50%) Effect = +16% • Assumption: they would have grown at the same pace • Not sure…
Matching • We compare pairs of 2 individuals for which the values taken by ALL variables are the same
Matching • Variation: Propensity Score Matching: all the variables do not need to be exactly the same, but you look for individuals which have the same “profile” • Problems • This matching method requires a big dataset: find pairs on a sufficient number of variables • What about unobservables?
An example • Case study: US Congress elections, 2002 • 60 000 phone calls to potential voters to encourage them to vote; 25 000 reached • Outcome: did they actually go vote? • 1st method: compare the 25 000 (reached) vs. the 35 000 (not reached) • 2nd method: introduce co-variates in a regression • 3rd method: introduce baseline data (vote in 1998) • 4th method: do a matching
1st comparison: we suspect a selection bias Selection Bias: differences in observable / unobservable characteristics → differences in outcome not due to the treatment 13
For more details, please read Case Study: “Get out the vote? Do phone calls encourage voting” under References