Improving Education through Accountability and Evaluation: Lessons from Around the World

1. Building Evidence that Counts: Evaluating Career Pathways Interventions for Disconnected Youth and Adults in the U.S. David Fein Abt Associates Inc david_fein@abtassoc.com Improving Education through Accountability and Evaluation: Lessons from Around the World Rome, Italy 3 October, 2012

2. Acknowledgments • Work on Innovative Strategies for Increasing Self-Sufficiency (ISIS) evaluation sponsored by Administration for Children and Families, U.S. Department of Health and Human Services with major support from the Open Society Foundations, additional support from Joyce and Kresge foundations • Presentation represents the author’s views only • Paper available at: http://www.projectisis.org/wp-content/uploads/2012/05/ISIS-Career-Pathways-Framework-OPRE-2012-30_5-16-12.pdf

3. Introduction • Large population of adults and youth in the U.S. with low skills, little/no post-secondary education, and poor prospects for self-sufficiency • In the 1990s-2000s, national policies emphasized labor force attachment (versus education and training) • Evidence from social experiments influenced this emphasis • Consistent modest positive findings for employment-focused interventions; disappointing findings for education and training • Some argued human capital investments should be limited to younger ages • Findings characterize traditional educational approaches not well-suited to second-chance populations

4. The Ground Is Shifting • Increasing evidence of limits to strictly employment-focused strategies, especially in a weak economy • Recession hits low-skilled jobs hardest and accelerates technological shifts widening skills premiums • Improved understanding of challenges facing individuals seeking post-2ndary, and deficiencies in second- (and first-) chance systems • Outpouring of promising innovations addressing these issues • But these policies and systems remain small scale and highly fragmented… • Enter Career Pathways – a body of ideas and practices for integrating promising innovations promoted by a growing movement in the U.S. And, with work, a conceptual framework for building systematic, high-quality evidence on these ideas and practices • It’s not about work vs. education, it’s about joining the two • I.e., “We’ve got the… stuff, we’ve just got to bring it together!” (Obama, on the economy in general, 9-28-12)

5. Building Evidence on Career Pathways that Counts • Abt Associates, a global social policy research and technical assistance firm, is leading several national random assignment studies of promising career pathways interventions • The work has involved mapping out a conceptual framework and identifying priorities and opportunities for useful experiments • I will introduce this framework, show how it can help to organize a wide range of studies, and discuss some ways we are designing these projects so that their findings will count

6. Key Career Pathways Ideas • Address the wide range of skills and needs of second-chance populations • Create manageable, well-articulated training steps • Provide credentials valued in high demand occupations/sectors • Build effective partnerships • E.g., public agencies (education, labor, human services); community colleges; community-based organizations; employers • Vs. K-12, more diverse actors, complexity • Basic ideas also being applied in strengthening career pathways in secondary education

7. V. BA+ Programs Upper-Skilled Jobs IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs III. Short-Term Certificate ProgramsEntry-Level Skilled Jobs II. Sectoral Bridge Programs Semi-Skilled Jobs I. Basic Bridge Programs The Basic Career Pathways Model Prospects for good-paying, stable employment Occupational, academic, and life skills

8. Program Inputs: Signature CP Service Strategies • Comprehensive assessment • Academic and non-academic skills • Basic and technical skills instruction • Modularization, contextualization, acceleration, flexible delivery, active learning • Supports • Pro-active advising and guidance, supplemental instruction, social supports, supportive services, financial assistance • Employment connections • During and after training

9. Initial Targeting & Placement Decisions Participant Characteristics Theory of Change for Career Pathways • Demographic • Educational • Economic Take First/ Next Step In Career Pathway/Lattice Increase Performance & Persistence in Training Foundational Academic Skills • Certificate/Diploma • 2- year, 4- year Degree Occupational Skills Comprehensive Assessment TO NEXT STEP Improve Performance & Advancement in Jobs Psycho-social Factors Core Curriculum •  Earnings •  Benefits •  Job security Career Orientation and Knowledge Supports Employment Connections Resource Constraints Improve Other Outcomes Other Personal and Family Challenges • Income & assets • Child & adult well being • Local economic growth Contextual Factors: Institutional, Economic, Social Primary Outcomes Program Inputs Intermediate Outcomes

10. Illustration #1: Pima Community College Pathways to Healthcare Program (Arizona) • Basic elements • In partnership with county One-Stop, college provides training and credentials in 16 health care occupations, organized into 5 broad pathways (+ 3 broad levels), programs varying from 1-24 months. Targets low-income adults, substantially Latino population. Funded by federal HPOG program. • Signature strategies • Instruction: Upfront assessment and training planning, w/10-week contextualized basic skills development program if below required entry skill levels. Many programs offer streamlined/compressed formats, translate “clock-hours” at lower levels to prerequisites to higher levels. • Supports: More intensive and coordinated case management (via One-Stop) and academic advising and services (via college), financial support covers training and related expenses. • Employment: Outreach to engage local employers in training, internships, employment; use One-Stop services; MOUs linking to various county and state agencies.

11. Illustration #2: Year Up (National) • Basic elements • National non-profit operating in 8 urban areas. Partners with community colleges and major employers in financial services and information technology. Targets disadvantaged youth, 18-24, with high school diploma/equivalent. • Signature strategies • Instruction: Six-month “learning development” phase provides customized training in technical and contextualized basic and professional skills (e.g., communication, behavior). Earn 14+ college credits through agreement w/local colleges. • Supports: “High support, high feedback” environment—learning communities, pro-active counseling, structured group sessions, supportive services, and weekly performance-based stipend of up to $260 throughout the year. Post-program follow-up and engagement. • Employment: Second six-month internship phase places students in entry-level financial services/IT positions with major employers, who contribute about half of total program costs per participant.

12. Illustrative Evidence on More Explicit Career Pathways Approaches: Comprehensive, well-targeted “first step” training programs can have substantial impacts Sectoraltraining (Maguire et al. 2010) 3 experienced CBOs provide customized short-term training in varied high-demand fields Disadvantaged adults, HS+, careful screening (n=1,014) Year Up (Roder & Elliott 2011) National org provides 6 months customized training + 6 month paid internship in IT and finance sectors Disadvantaged urban youth 18-24, HS+ Program articulates w/local colleges, completers earn 14+ credits Results from small initial experiment (n=164)

13. Examples of Recent Experimental Findings on Signature CP Strategies • Learning communities • As bridge program (Fein et al. 2006) • By linking courses (Sommo et al. 2012; Visher et al. 2012) • More intensive, specialized personal guidance and support • Coaching (Bettinger & Baker 2011) • Help with financial aid applications (Bettinger et al. 2012) • Student success courses + support (Weiss et al. 2011) • Increased financial support • Performance-based scholarships (Patel & Richburg-Hayes 2012) • Unconditional needs-based grants (Goldrick-Rab et al. 2011) • Psycho-social interventions • E.g., implicit theories of intelligence, normalizing worries about fitting in and success (Yeager & Walton 2011)

14. Designing Evaluations that Matter Evaluations should be designed in ways that invite practitioners to make use of the results and adopt solid practices based on evidence. We need to recognize the role of motivated program leaders at the center of evaluation efforts, to ensure that these efforts advance program theory and practice, rather than merely fulfilling a funder request. Research tells us that organizations and their leaders need to own and trust information in order to use it. Partnership on the ground between practitioners and evaluators, along with the long-term support of committed public and private funders, is indispensable if the goal is to deliver evaluations that actually improve program quality and effectiveness. • Public/Private Ventures, Priorities for a New Decade: Making (More) Social Programs Work (Better), 2011. Principles • Elucidate and address theories of program effectiveness • Develop strong common metrics that will be easily understood and widely accepted • Design to anticipate interest in scaling and replicating successful approaches • Foster appreciation for technical rigor (e.g., experimental designs) • Produce evaluation products useful to policymakers and practitioners • Emphasis on active interaction: evaluators-practitioner, practitioner-practitioner

15. Example: The ISIS Evaluation • First national random assignment study of U.S. career pathways programs • Separate tests of nine relatively comprehensive career pathways programs, with implementation and cost-benefit studies • Central goals • Assess overall effectiveness of each program • Understand “why and under what conditions?” • Work within well-specified theory of change • Strong focus on impacts on exposure to program inputs, intervening outcomes, heterogeneity • Non-experimental analysis of mediators of impacts on primary outcomes • Strong emphasis on collaboration helps ensure findings will count…

16. Collaboration Increases Chances that Findings Will Matter • Extensive stakeholder outreach + literature review identify career pathways focus in ISIS… • Early and ongoing exchanges foster interest and create positive climate for experiments among potential sites • Working w/sites as partners w/complementary expertise and interests • Building an ongoing “learning community” • Learn from experts on the team, peer-to-peer exchange, participation in external forums • Help plan and execute dissemination strategies • Maintain and extend relationships with wider stakeholder network • Including private and public funding agencies, policy makers, professional associates, advocates, practitioners, • Through meetings, website, public webinars, varied publications • On diverse interests: related projects, substantive topics/directions in ISIS, evaluation findings, evaluation products, • So that findings aren’t limited to a small number of reports, but shared in dynamic, diversified ways with engaged audiences

17. Summary • Career pathways framework useful in program and evaluation design, organizing emerging findings • Genuine partnerships with practitioners and other stakeholders key to designing evaluations that matter • Thank you!