Randomized evaluations applications
Download
1 / 64

Randomized Evaluations: Applications - PowerPoint PPT Presentation


  • 61 Views
  • Uploaded on

Randomized Evaluations: Applications. Kenny Ajayi September 22, 2008. Global Poverty and Impact Evaluation. Review. Causation and Correlation (5 factors) X → Y Y → X X → Y and Y → X Z → X and Z → Y X ‽ Y . Review. Causation and Correlation (5 factors) X → Y causality

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Randomized Evaluations: Applications' - shiro


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Randomized evaluations applications

Randomized Evaluations:Applications

Kenny Ajayi

September 22, 2008

Global Poverty and Impact Evaluation


Review
Review

  • Causation and Correlation (5 factors)

    • X → Y

    • Y → X

    • X → Y and Y → X

    • Z → X and Z → Y

    • X ‽ Y


Review1
Review

  • Causation and Correlation (5 factors)

    • X → Y causality

    • Y → X reverse causality

    • X → Y and Y → X simultaneity

    • Z → X and Z → Y omitted variables / confounding

    • X ‽ Y spurious correlation

  • Reduced-form relationship between X and Y


Review2
Review

Exogenous


Review3
Review

Exogenous

Endogenous


Review4
Review

Exogenous

Endogenous

External Validity


Review5
Review

Exogenous

Endogenous

External Validity

Natural Experiment


Practical applications
Practical Applications

  • Noncompliance

  • Cost Benefit Analysis

  • Theory and Mechanisms


Kling liebman and katz 2007
Kling, Liebman and Katz (2007)

  • Do neighborhoods affect their residents?



Do neighborhoods affect their residents1
Do neighborhoods affect their residents?

  • Positively

    • Social connections, role models, security, community resources


Do neighborhoods affect their residents2
Do neighborhoods affect their residents?

  • Positively

    • Social connections, role models, security, community resources

  • Negatively


  • Do neighborhoods affect their residents3
    Do neighborhoods affect their residents?

    • Positively

      • Social connections, role models, security, community resources

  • Negatively

    • Discrimination, competition with advantaged peers


  • Do neighborhoods affect their residents4
    Do neighborhoods affect their residents?

    • Positively

      • Social connections, role models, security, community resources

  • Negatively

    • Discrimination, competition with advantaged peers

  • Not at all


  • Do neighborhoods affect their residents5
    Do neighborhoods affect their residents?

    • Positively

      • Social connections, role models, security, community resources

  • Negatively

    • Discrimination, competition with advantaged peers

  • Not at all

    • Only family influences, genetic factors, individual human capital investments or broader non-neighborhood environment matter



  • Do neighborhoods affect their residents7
    Do neighborhoods affect their residents?

    • What is Y?

      • Adults

        • Self sufficiency

        • Physical & mental health

      • Youth

        • Education (reading/math test scores)

        • Physical & mental health

        • Risky behavior


    Kling liebman and katz 20071
    Kling, Liebman and Katz (2007)

    • Do neighborhoods affect their residents?

      • X → Y causality

      • Y → X reverse causality

      • X → Y and Y → X simultaneity

      • Z → X and Z → Y omitted variables / confounding

      • X ‽ Y spurious correlation


    Neighborhood effects
    Neighborhood Effects

    • Natural Experiment


    Neighborhood effects1
    Neighborhood Effects

    • Natural Experiment

      • Hurricane Katrina

    • Can we randomly assign people to live in different neighborhoods?


    Neighborhood effects2
    Neighborhood Effects

    • Natural Experiment

      • Hurricane Katrina

    • Can we randomly assign people to live in different neighborhoods?

      • Sort of…


    Moving to opportunity
    Moving to Opportunity

    • Solicit volunteers to participate

    • Randomly assign each volunteer an option

      • Control

      • Section 8

      • Section 8 with restriction (low poverty neighborhood)

    • Volunteers decide whether to move

      • Compliers: use voucher

      • Non-compliers: don’t move



    Non random participation
    Non-random Participation

    • Why is this OK?

      • Volunteers presumably care about their neighborhood environment, so should be the target when thinking about uptake for the use of housing vouchers

        BUT

      • Results might not generalize to other populations with different characteristics


    Treatment and compliance
    Treatment and Compliance

    • With perfect compliance:

      • Pr(X=1│Z=1)=1

      • Pr(X=1│Z=0)=0

    • With imperfect compliance:

      • 1>Pr(X=1│Z=1)>Pr(X=1│Z=0)>0 

        Where X - actual treatment

        Z - assigned treatment


    Some econometrics
    Some Econometrics

    • Intention-to-treat (ITT) Effects

      • Differences between treatment and control group means


    Some econometrics1
    Some Econometrics

    • Intention-to-treat (ITT) Effects

      • Differences between treatment and control group means

    ITT = E(Y|Z = 1) – E(Y|Z = 0)

    Y - outcome

    Z - assigned treatment


    Some econometrics2
    Some Econometrics

    • Intention-to-treat (ITT) Effects

      • Writing this relationship in mathematical notation

    Y = Zπ1 + Wβ1 + ε1

    Individual wellbeing (Y) depends on whether or not the individual is assigned to treatment (Z), baseline characteristics (W), and whether or not the individual got lucky (ε)

    π1- effect of being assigned to treatment group

    = (effect of actually moving) x (compliance rate)


    Some econometrics3
    Some Econometrics

    • Effect of Treatment on the Treated (TOT)

      • Differences between treatment and control group means

        Differences in compliance for treatment and control groups


    Some econometrics4
    Some Econometrics

    • Effect of Treatment on the Treated (TOT)

      • Differences between treatment and control group means

        Differences in compliance for treatment and control groups

      • TOT = “Wald Estimator” = E(Y|Z = 1) – E(Y|Z = 0)

        E(X|Z = 1) – E(X|Z = 0)

        Y - outcome

        Z - assigned treatment

        X - actual treatment (i.e. compliance)


    Some more econometrics
    Some More Econometrics

    • Estimate TOT using instrumental variables

      • Use offer of a MTO voucher as an instrument for MTO voucher use


    Some more econometrics1
    Some More Econometrics

    • Estimate TOT using instrumental variables

      • Use offer of a MTO voucher as an instrument for MTO voucher use. In mathematical notation:

        (1) X = Zρd + Wβd + εd

      • Predict whether people use voucher (X), given their assigned treatment (Z) and observable characteristics (W)


    Some more econometrics2
    Some More Econometrics

    • Estimate TOT using instrumental variables

      • Use offer of a MTO voucher as an instrument for MTO voucher use. In mathematical notation:

        (1) X = Zρd + Wβd + εd

        (2)Y = Xγ2 + Wβ2 + ε2

      • Predict whether people use voucher (X), given their assigned treatment (Z) and observable characteristics (W)

      • Estimate how much individual wellbeing (Y) depends on predicted compliance (X) and observable characteristics (W)


    Some econometrics5
    Some Econometrics

    • Effect of Treatment on the Treated (TOT)

      • Writing the reduced-form relationship in mathematical notation

    Y = Xγ2 + Wβ2 + ε2

    Individual wellbeing (Y) depends on whether or not the individual complies with treatment (X), baseline characteristics (W), and whether or not the individual got lucky (ε)

    Note:γ2-effect of actuallymoving (i.e. compliance)

    = effect of being assigned to treatment group compliance rate



    Moving to opportunity3
    Moving to Opportunity

    INTENTION TO TREAT


    Moving to opportunity4
    Moving to Opportunity

    EFFECT OF TREATMENT ON THE TREATED


    Results
    Results

    • Positive effects

      • Teenage girls

      • Adult mental health

    • Negative effects

      • Teenage boys

    • No effects

      • Adult economic self-sufficiency & physical health

      • Younger children

    • Increasing effects for lower poverty rates


    Limitations
    Limitations

    • Can’t fully separate relocation effect from neighborhood effect

    • Can’t observe spillovers into receiving neighborhoods


    Some thoughts
    Some Thoughts

    • We can still estimate impacts in cases where we don’t have full compliance

    • Policies sometimes target people who are different from the general population (i.e. those more likely to take up a program)

      BUT

    • We have to be careful about generalizing these results for other populations (external validity)


    Practical applications1
    Practical Applications

    Noncompliance

    Cost Benefit Analysis

    Theory and Mechanisms


    Duflo kremer robinson 2008
    Duflo, Kremer & Robinson (2008)

    • Why don’t Kenyan farmers use fertilizer?


    Duflo kremer robinson 20081
    Duflo, Kremer & Robinson (2008)

    • Why don’t Kenyan farmers use fertilizer?

      • Low returns

      • High cost

      • Preference for traditional farming methods

      • Lack of information


    Is fertilizer use profitable
    Is Fertilizer Use Profitable?

    • What is Y (outcome)?

      • Crop yield

      • Rate of return over the season

    • What is X (observed)?

      • Fertilizer

      • Hybrid seed


    Is fertilizer use profitable1
    Is Fertilizer Use Profitable?

    • What is Z (unobserved)?


    Is fertilizer use profitable2
    Is Fertilizer Use Profitable?

    • What is Z (unobserved)?

      • Farmer experience/ability

      • Farmer resources

      • Weather

      • Soil quality

      • Market conditions


    Correlation or causation
    Correlation or Causation?

    Fertilizer Use (X) and Crop Yield (Y)


    Correlation or causation1
    Correlation or Causation?

    Fertilizer Use (X) and Crop Yield (Y)

    • X → Y causality

    • Y → X reverse causality

    • X → Y and Y → X simultaneity

    • Z → X and Z → Y omitted variables / confounding

    • X ‽ Y spurious correlation


    Fertilizer experiment
    Fertilizer Experiment

    • How could we test the effect of fertilizer use on crop yield using randomization?


    Fertilizer experiment1
    Fertilizer Experiment

    • Randomly select participating farmers

    • Measure 3 adjacent equal-sized plots on each farm

    • Randomly assign farming package for each plot

      • Top dressing fertilizer (T1)

      • Fertilizer and hybrid seed (T2)

      • Traditional seed (C)

    • Weigh each plot’s yield at harvest


    Fertilizer experiment2
    Fertilizer Experiment

    • Calculate rate of return over the season:

      value of value of

      treatment - comparison - value of

      plot output plot output input

      RR = ----------------------------------------------------------- - 100

      value of input





    Some thoughts1
    Some Thoughts

    • More comprehensive package increased yield

      BUT

    • Lower cost package had highest rate of return

    • Lab results don’t always apply in the field

    • Think about the COSTS and BENEFITS


    Practical applications2
    Practical Applications

    Noncompliance

    Cost Benefit Analysis

    Theory and Mechanisms


    Karlan and zinman 2007
    Karlan and Zinman (2007)

    • What is the optimal loan contract?


    Objectives
    Objectives

    • Profitability

      • Gross Revenue

      • Repayment

    • Targeted Access

      • Poor households

      • Female microentrepreneurs

    • Sustainability


    Theory of change
    Theory of Change

    Loan Profits

    contract

    • Mechanisms

      • How do we expect a change to affect the outcome?

      • What assumptions are we making?

      • Can we test if they are true?


    Loan offer experiment
    Loan Offer Experiment

    • What is Y?

      • Take-up

      • Loan size

      • Default

  • What is X?

    • Interest rate

    • Maturity

  • How sensitive are borrowers to interest rates and maturities?



  • Results4
    Results

    • Downward sloping demand curves

    • Loan size more responsive to loan maturity than to interest rate


    Some limitations
    Some Limitations

    • Again, results apply only to people who had previously borrowed from the lender.

    • Elasticities apply to direct mail solicitations and might not hold for other loan offer methods


    Final thoughts
    Final Thoughts

    • Randomized evaluation can have many applications beyond merely looking at reduced-form relationships between

      X and Y