Agenda
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

Agenda: PowerPoint PPT Presentation


  • 81 Views
  • Uploaded on
  • Presentation posted in: General

Agenda:. Block Watch: Random Assignment, Outcomes, and indicators Issues in Impact and Random Assignment: Youth Transition Demonstration Who is randomized? Sample size, power, and effect size Who’s in the average?. Block Watch: Random Assignment, Outcomes, and Indicators.

Download Presentation

Agenda:

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Agenda

Agenda:

Block Watch: Random Assignment, Outcomes, and indicators

Issues in Impact and Random Assignment: Youth Transition Demonstration

Who is randomized?

Sample size, power, and effect size

Who’s in the average?


Block watch random assignment outcomes and indicators

Block Watch: Random Assignment, Outcomes, and Indicators

  • What random assignment protocol would you use to assess the impacts of Block Watch?

  • What are the strengths and weaknesses of your approach?

  • What are the key outcomes you want to assess? What are indicators for those?


Youth transition demonstration evaluation plan

Youth Transition Demonstration Evaluation Plan:

  • Background on YTD evaluation plan

  • The basics of Impact size and significance

  • Power and sample size

  • No Shows/ Intent to Treat vs. Treatment on the Treated

  • Multiple Comparisons

  • Regression adjusted comparisons


Youth transition demonstration

Youth Transition Demonstration:

  • Targets Youth receiving disability payments to help in transition to adult life and employment

  • Goals: increase earnings, decrease costs, facilitate transition to self-sufficiency

  • Six program sites with variation in programs

  • Services:

    • Waiver of benefit decrease with earnings

    • Education, Job training, work placements

    • Case management, counseling, referral to services


Ytd evaluation

YTD Evaluation:

  • Selected 6 sites for demonstration and evaluation

  • Intervention built on research from past programs and evaluations

  • Randomly assigned youth to treatment or control

  • Large sample sizes to allow identification of smaller effects and sub-group effects

  • Process and Impact Evaluation

  • Data collected from administrative files, surveys before and after program

  • Advisory group of experts


Sampling

Sampling:

  • Why did they divide the list of potential participants (sampling frame) into groups of 10 for contact?

  • Why did they randomize 55 percent to the treatment?

  • Why get pre-intervention characteristics if they are randomly assigning groups?


Agenda

Comparisons may be: -over time -across intervention groups with and without program;levels of intervention (“dosage”)

Impact here!


Agenda

Statistical significance:

When can we rule out having an impact IF there is no impact?

Compare 2 means from independent samples:

Means: Proportions:

Pooled sample variance:


Agenda

Compare 2 means from independent samples:

Means: Proportions:

Pooled sample variance:


Agenda

Compare 2 means from independent samples:

Means: Proportions:

Pooled sample variance:


Agenda

Compare 2 means from independent samples:

Means: Proportions:

Pooled sample variance:


So it s easier to say impact is real not just randomness if

So, it’s easier to say impact is “real” (not just randomness) if:

  • Size of impact is larger

  • Variation in outcomes is small (S)

  • Sample sizes are larger

    Same factors figure into deciding how big a sample we need to find the effect if it’s there! [Power, sample size, minimally detectable effects]


Agenda

Power and sample size: Given randomness, what % of time will you be able to rule out the null, IF it is NOT true (there IS an impact)?How big a sample size do you need to rule out NO effect if the program DOES have an impact? (Rossi et al p.312)


Online calculators for sample size and power

Online Calculators for Sample size and Power:

  • Sample size:

    • http://www.dssresearch.com/toolkit/sscalc/size_a2.asp

    • http://www.dssresearch.com/toolkit/sscalc/size_p2.asp

  • Power:

    • http://www.dssresearch.com/toolkit/spcalc/power_a2.asp

    • http://statpages.org/proppowr.html


Agenda

Minimum Detectable Impacts:What are the smallest effects you will be able to detect given n and predicted S?


Adjustments to impact assessment

Adjustments to impact assessment:

  • Regression adjusted impacts decrease S and increase power by controlling for “noise” using baseline characteristics

  • Multiple Comparisons are a problem because randomness happens if you look long enough!

    • MDRC picked “primary outcomes”

    • Use adjustments to account for multiple comparisons


Showing estimated impacts over time in program

Showing estimated impacts over time in program


Who s in the average

Who’s in the average?

  • “No shows” in treatment group didn’t get any services

    • Unlikely to be similar to “shows”

    • If drop, then may overstate potential impacts

  • “Intent to Treat” outcomes include outcomes for no-shows

  • “Treatment on the Treated” outcomes do not include no-shows

  • Non-response to follow-up surveys could bias impact assessments

    • Use administrative data available for all for key outcomes

    • Put resources into follow up to minimize non-response

    • Construct weights to make survey sample estimates comparable to baseline sample


Lessons from summary

Lessons from Summary:

  • Randomization is hard

  • Need to use power analysis to choose target sample sizes

  • Even randomization may not give comparable baseline characteristics

  • Regression may increase comparability and precision

  • Worry about who we have outcome information for (both control and treatment)


  • Login