The Value of Random Assignment Impact Evaluations for Youth-Serving Interventions?

1. The Value of Random Assignment Impact Evaluations for Youth-Serving Interventions? Notes from Career Academy Research and Practice James Kemple Senior Fellow, MDRC www.mdrc.org

2. What Can Career Academy Research and Practice Offer Evidence-Based Policy? Practice • 34-year track record of implementation, planned expansion, and efforts at continuous improvement • Intervention with goals and core features aligned with important problems in high schools and prominent policy options Research • 25 years of non-experimental research and a commitment to learning what works • 10-year random assignment field experiment involving 9 sites, over 1,700 students, and 8-years of follow-up • Positive effects on labor market outcomes without compromising on academic goals

3. Context for Impact Evaluations • Learning “what works” is a long-term and cumulative process • Questions drive methodology, not the reverse • Multiple questions require multiple methods • Must balance research ambition against operational and political realities • Knowledge-building should be an integral part of policy development and continuous improvement, not an add-on or after-thought

4. Why Conduct Impact Evaluations? • Outcomes vs. Impacts • Outcome: Measure of individual or group behavior, attitudes, achievement, labor market participation, an so on. • Impact: The effect on an intervention on an outcome: the difference between outcome for program group and outcome for counterfactual. • Outcome-focused studies risk getting the wrong answer to the right question • Outcome standards risk awarding programs: • based on who they serve, rather than what they do • that operate under promising conditions, rather than use promising practices

5. National Averages for Similar Students in Similar Schools 100 Evaluation Sample 90 8.6 80 70 60 Percent Graduating On-Time 50 80.4 40 72.9 72.2 63.3 30 48.6 20 10 0 Academy Non-Academy Career/Tech. General Academic Academy Judging Program Impacts:High Outcomes/No Impact Note: National average estimates are adjusted to represent a sample with the same background characteristics as those in the Evaluation Sample.

6. 100 100 100 90 90 90 85 85 84 ) ) 80 80 80 ) 72 71 70 70 70 60 60 60 53 50 50 50 Graduation Rate (% Graduation Rates (% Graduation Rates (% 40 40 40 30 30 30 20 20 20 10 10 10 0 0 0 Program A Program B Program C Academy Group Control Group Judging Program Impacts:Adding value vs. starting with strong context

7. Career Academy Impacts on Average Earnings Impacts

8. Guiding Principles for Impact Evaluations • Random assignment may be the “gold standard” but it is not the “Philosopher’s Stone” (i.e., won’t extend life or answer every important question.) • Questions drive methodology, not the reverse • Because evaluations involve multiple questions they require multiple methods • Implementing methodologies requires balancing research ambition against operational realities • Strong research designs cannot compensate for weak treatments

9. Conditions for Random Assignment • Priority Question: What is the impact? • Ethical and legal standards • No denial of services to which otherwise entitled • No reduction in expected service levels • Informed consent and data confidentiality • Operational Realities • Collaboration between researchers and program managers • Structured process for program entry or access to resources • Excess demand: more eligible applicants than available program slots or resources • Fair method for allocating scarce resources • Opportunity for a “fair test” of the intervention

10. Conditions for a “Fair Test” • Strong contrast with “status quo” • Implementation of program being tested • Participant exposure to program services • Well-understood alternative to program service • High quality methods for answering questions about why programs are effective (or not) and for whom • Dissemination of findings about what works and what does not work

11. Implications of Career Academies Evaluation • Random assignment provided findings that could not have been obtained with other designs. • Increased investments in career-related experiences during high school can improve post-secondary labor market prospects. • Career Academies serve as viable pathway to post-secondary education, but not necessarily better than other opportunities. • Career Academies demonstrate feasibility of accomplishing goals of school-to-career and career technical education without compromising on academic goals.