David evans impact evaluation cluster aftrl
Download
1 / 27

Impact Evaluation Methods: Difference in difference & Matching - PowerPoint PPT Presentation


  • 914 Views
  • Updated On :

David Evans Impact Evaluation Cluster, AFTRL. Africa Program for Education Impact Evaluation. Impact Evaluation Methods: Difference in difference & Matching. Slides by Paul J. Gertler & Sebastian Martinez. AFRICA IMPACT EVALUATION INITIATIVE, AFTRL. Measuring Impact. Randomized Experiments

Related searches for Impact Evaluation Methods: Difference in difference & Matching

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Impact Evaluation Methods: Difference in difference & Matching' - cricket


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
David evans impact evaluation cluster aftrl l.jpg

David Evans

Impact Evaluation Cluster, AFTRL

Africa Program for Education Impact Evaluation

Impact Evaluation Methods: Difference in difference & Matching

Slides by Paul J. Gertler & Sebastian Martinez

AFRICA IMPACT EVALUATION INITIATIVE, AFTRL


Measuring impact l.jpg
Measuring Impact

  • Randomized Experiments

  • Quasi-experiments

    • Randomized Promotion – Instrumental Variables

    • Regression Discontinuity

    • Double differences (Diff in diff)

    • Matching


Case 5 diff in diff l.jpg
Case 5: Diff in diff

  • Compare change in outcomes between treatments and non-treatment

    • Impact is the difference in the change in outcomes

  • Impact = (Yt1-Yt0) - (Yc1-Yc0)


Slide4 l.jpg

Average Treatment Effect

Outcome

Treatment Group

Control Group

Time

Treatment


Slide5 l.jpg

Outcome

Time

Treatment

Measured effect without pre-measurement

Treatment Group

Control Group


Slide6 l.jpg

EstimatedAverage Treatment Effect

Treatment Group

Control Group

Outcome

Average Treatment Effect

Time

Treatment


Diff in diff l.jpg
Diff in diff

  • What is the key difference between these two cases?

  • Fundamental assumption that trends (slopes) are the same in treatments and controls (sometimes true, sometimes not)

  • Need a minimum of three points in time to verify this and estimate treatment (two pre-intervention)


Slide8 l.jpg

Average Treatment Effect

Outcome

Treatment Group

Third

observation

Control Group

Second

observation

First

observation

Time

Treatment


Examples l.jpg
Examples

  • Two neighboring school districts

    • School enrollment or test scores are improving at same rate before the program (even if at different levels)

    • One receives program, one does not

    • Neighboring _______


Case 5 diff in diff10 l.jpg

Mean change

CPC

Case 5 - Diff in Diff

Linear Regression

Multivariate Linear Regression

27.66**

25.53**

Estimated Impact on CPC

(2.68)

(2.77)

** Significant at 1% level

Case 5: Diff in Diff

Case 5 - Diff in Diff

Not Enrolled

Enrolled

t-stat

8.26

35.92

10.31


Impact evaluation example summary of results l.jpg

Case 2 -

Case 4 -

Case 1 - Before

Case 3 -

Case 5 - Diff in

Enrolled/Not

Regression

and After

Randomization

Diff

Enrolled

Discontinuity

Multivariate

Multivariate

Multivariate

Multivariate

Linear

Multivariate Linear

Linear

Linear

Linear

Regression

Regression

Regression

Regression

Regression

Estimated Impact

34.28**

-4.15

29.79**

30.58**

25.53**

on CPC

(2.11)

(4.05)

(3.00)

(5.93)

(2.77)

** Significant at 1% level

Impact Evaluation Example –Summary of Results

Impact Evaluation Example –Summary of Results


Example l.jpg
Example

  • Old-age pensions and schooling in South Africa

    • Eligible if household member over 60

    • Not eligible if under 60

      • Used household with member age 55-60

    • Pensions for women and girls’ education


Measuring impact13 l.jpg
Measuring Impact

  • Randomized Experiments

  • Quasi-experiments

    • Randomized Promotion – Instrumental Variables

    • Regression Discontinuity

    • Double differences (Diff in diff)

    • Matching


Matching l.jpg
Matching

  • Pick the ideal comparison group that matches the treatment group from a larger survey.

  • The matches are selected on the basis of similarities in observed characteristics.

    • For example?

  • This assumes no selection bias based on unobserved characteristics.

    • Example: income

    • Example: entrepreneurship

      Source: Martin Ravallion


Propensity score matching psm l.jpg
Propensity-Score Matching (PSM)

  • Controls: non-participants with same characteristics as participants

    • In practice, it is very hard. The entire vector of X observed characteristics could be huge.

  • Match on the basis of the propensity score

    P(Xi) = Pr (participationi=1|X)

    • Instead of aiming to ensure that the matched control for each participant has exactly the same value of X, same result can be achieved by matching on the probability of participation.

    • This assumes that participation is independent of outcomes given X (not true if important unobserved outcomes are affecting participation)


Steps in score matching l.jpg
Steps in Score Matching

  • Representative & highly comparable survey of non-participants and participants.

  • Pool the two samples and estimate a logit (or probit) model of program participation:

    Gives the probability of participating for a person with X

  • Restrict samples to assure common support (important source of bias in observational studies)

    For each participant find a sample of non-participants that have similar propensity scores

    Compare the outcome indicators. The difference is the estimate of the gain due to the program for that observation.

    Calculate the mean of these individual gains to obtain the average overall gain.


Slide17 l.jpg

Density of scores for participants

Density

Region of common support

High probability of participating given X

0

1

Propensity score


Steps in score matching18 l.jpg
Steps in Score Matching

  • Representative & highly comparable survey of non-participants and participants.

  • Pool the two samples and estimate a logit (or probit) model of program participation:

    Gives the probability of participating for a person with X

  • Restrict samples to assure common support (important source of bias in observational studies)

  • For each participant find a sample of non-participants that have similar propensity scores

  • Compare the outcome indicators. The difference is the estimate of the gain due to the program for that observation.

  • Calculate the mean of these individual gains to obtain the average overall gain.


Psm vs an experiment l.jpg
PSM vs an experiment

  • Pure experiment does not require the untestable assumption of independence conditional on observables

  • PSM requires large samples and good data


Lessons on matching methods l.jpg
Lessons on Matching Methods

  • Typically used for IE when neither randomization, RD or other quasi-experimental options are not possible (i.e. no baseline)

    • Be cautious of ex-post matching:

      • Matching on variables that change due to participation (i.e., endogenous)

      • What are some variables that won’t change?

  • Matching helps control for OBSERVABLE differences


More lessons on matching methods l.jpg
More Lessons on Matching Methods

  • Matching at baseline can be very useful:

    • Estimation:

      • Combine with other techniques (i.e. diff in diff)

      • Know the assignment rule (match on this rule)

    • Sampling:

      • Selecting non-randomized control sample

  • Need good quality data

    • Common support can be a problem


Case 7 matching l.jpg

Case 7 - PROPENSITY SCORE: Pr(treatment=1)

Variable

Coef.

Std. Err.

-0.03

0.00

Age Head

-0.05

0.01

Educ Head

-0.02

0.00

Age Spouse

-0.06

0.01

Educ Spouse

0.42

0.04

Ethnicity

-0.23

0.07

Female Head

Constant

1.6

0.10

P-score Quintiles

Quintile 1

Quintile 2

Quintile 3

Quintile 4

Quintile 5

T

C

t-score

T

C

t-score

T

C

t-score

T

C

t-score

T

C

t-score

Xi

68.04

67.45

-1.2

53.61

53.38

-0.51

44.16

44.68

1.34

37.67

38.2

1.72

32.48

32.14

-1.18

Age Head

1.54

1.97

3.13

2.39

2.69

1.67

3.25

3.26

-0.04

3.53

3.43

-0.98

2.98

3.12

1.96

Educ Head

55.95

55.05

-1.43

46.5

46.41

0.66

39.54

40.01

1.86

34.2

34.8

1.84

29.6

29.19

-1.44

Age Spouse

1.89

2.19

2.47

2.61

2.64

0.31

3.17

3.19

0.23

3.34

3.26

-0.78

2.37

2.72

1.99

Educ Spouse

0.16

0.11

-2.81

0.24

0.27

-1.73

0.3

0.32

1.04

0.14

0.13

-0.11

0.7

0.66

-2.3

Ethnicity

0.19

0.21

0.92

0.42

0.16

-1.4

0.092

0.088

-0.35

0.35

0.32

-0.34

0.008

0.008

0.83

Female Head

Case 7: Matching


Case 7 matching23 l.jpg

Case 7 - Matching

Linear Regression

Multivariate Linear Regression

1.16

7.06+

Estimated Impact on CPC

(3.59)

(3.65)

** Significant at 1% level, + Significant at 10% level

Case 7: Matching


Impact evaluation example summary of results24 l.jpg

Case 2 -

Case 4 -

Case 1 - Before

Case 3 -

Case 5 - Diff in

Case 6 - IV

Case 7 -

Enrolled/Not

Regression

and After

Randomization

Diff

(TOT)

Matching

Enrolled

Discontinuity

Multivariate

Multivariate

Multivariate

Multivariate

Multivariate

Linear

Multivariate Linear

Linear

Linear

Linear

Linear

Regression

Regression

Regression

Regression

Regression

2SLS

Regression

Estimated Impact

34.28**

-4.15

29.79**

30.58**

25.53**

30.44**

7.06+

on CPC

(2.11)

(4.05)

(3.00)

(5.93)

(2.77)

(3.07)

(3.65)

** Significant at 1% level

Impact Evaluation Example –Summary of Results


Measuring impact25 l.jpg
Measuring Impact

  • Experimental design/randomization

  • Quasi-experiments

    • Regression Discontinuity

    • Double differences (Diff in diff)

    • Other options

      • Instrumental Variables

      • Matching

    • Combinations of the above


Remember l.jpg
Remember…..

  • Objective of impact evaluation is to estimate the CAUSAL effect of a program on outcomes of interest

  • In designing the program we must understand the data generation process

    • behavioral process that generates the data

    • how benefits are assigned

  • Fit the best evaluation design to the operational context