slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Improving Education through Accountability and Evaluation: Lessons from Around the World PowerPoint Presentation
Download Presentation
Improving Education through Accountability and Evaluation: Lessons from Around the World

Loading in 2 Seconds...

play fullscreen
1 / 17

Improving Education through Accountability and Evaluation: Lessons from Around the World - PowerPoint PPT Presentation

  • Uploaded on

Building Evidence that Counts: Evaluating Career Pathways Interventions for Disconnected Youth and Adults in the U.S. David Fein Abt Associates Inc Improving Education through Accountability and Evaluation: Lessons from Around the World Rome, Italy 3 October, 2012.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Improving Education through Accountability and Evaluation: Lessons from Around the World' - syshe

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Building Evidence that Counts: Evaluating Career Pathways Interventions for Disconnected Youth and Adults in the U.S.

David Fein

Abt Associates Inc

Improving Education through Accountability and

Evaluation: Lessons from Around the World

Rome, Italy

3 October, 2012

  • Work on Innovative Strategies for Increasing Self-Sufficiency (ISIS) evaluation sponsored by Administration for Children and Families, U.S. Department of Health and Human Services with major support from the Open Society Foundations, additional support from Joyce and Kresge foundations
  • Presentation represents the author’s views only
  • Paper available at:
  • Large population of adults and youth in the U.S. with low skills, little/no post-secondary education, and poor prospects for self-sufficiency
  • In the 1990s-2000s, national policies emphasized labor force attachment (versus education and training)
  • Evidence from social experiments influenced this emphasis
    • Consistent modest positive findings for employment-focused interventions; disappointing findings for education and training
    • Some argued human capital investments should be limited to younger ages
  • Findings characterize traditional educational approaches not well-suited to second-chance populations
the ground is shifting
The Ground Is Shifting
  • Increasing evidence of limits to strictly employment-focused strategies, especially in a weak economy
  • Recession hits low-skilled jobs hardest and accelerates technological shifts widening skills premiums
  • Improved understanding of challenges facing individuals seeking post-2ndary, and deficiencies in second- (and first-) chance systems
  • Outpouring of promising innovations addressing these issues
  • But these policies and systems remain small scale and highly fragmented…
  • Enter Career Pathways – a body of ideas and practices for integrating promising innovations promoted by a growing movement in the U.S. And, with work, a conceptual framework for building systematic, high-quality evidence on these ideas and practices
    • It’s not about work vs. education, it’s about joining the two
    • I.e., “We’ve got the… stuff, we’ve just got to bring it together!” (Obama, on the economy in general, 9-28-12)
building evidence on career pathways that counts
Building Evidence on Career Pathways that Counts
  • Abt Associates, a global social policy research and technical assistance firm, is leading several national random assignment studies of promising career pathways interventions
  • The work has involved mapping out a conceptual framework and identifying priorities and opportunities for useful experiments
  • I will introduce this framework, show how it can help to organize a wide range of studies, and discuss some ways we are designing these projects so that their findings will count
key career pathways ideas
Key Career Pathways Ideas
  • Address the wide range of skills and needs of second-chance populations
  • Create manageable, well-articulated training steps
  • Provide credentials valued in high demand occupations/sectors
  • Build effective partnerships
    • E.g., public agencies (education, labor, human services); community colleges; community-based organizations; employers
    • Vs. K-12, more diverse actors, complexity
  • Basic ideas also being applied in strengthening career pathways in secondary education

V. BA+ Programs Upper-Skilled Jobs

IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs

III. Short-Term Certificate ProgramsEntry-Level Skilled Jobs

II. Sectoral Bridge Programs Semi-Skilled Jobs

I. Basic Bridge Programs

The Basic Career Pathways Model

Prospects for good-paying, stable employment

Occupational, academic, and life skills

program inputs signature cp service strategies
Program Inputs: Signature CP Service Strategies
  • Comprehensive assessment
    • Academic and non-academic skills
  • Basic and technical skills instruction
    • Modularization, contextualization, acceleration, flexible delivery, active learning
  • Supports
    • Pro-active advising and guidance, supplemental instruction, social supports, supportive services, financial assistance
  • Employment connections
    • During and after training

Initial Targeting & Placement Decisions

Participant Characteristics

Theory of Change for

Career Pathways

  • Demographic
  • Educational
  • Economic

Take First/ Next Step In Career Pathway/Lattice

Increase Performance & Persistence in Training

Foundational Academic Skills

  • Certificate/Diploma
  • 2- year, 4- year Degree

Occupational Skills

Comprehensive Assessment


Improve Performance & Advancement in Jobs

Psycho-social Factors

Core Curriculum

  •  Earnings
  •  Benefits
  •  Job security

Career Orientation and Knowledge


Employment Connections

Resource Constraints

Improve Other Outcomes

Other Personal and Family Challenges

  • Income & assets
  • Child & adult well being
  • Local economic growth

Contextual Factors: Institutional, Economic, Social

Primary Outcomes

Program Inputs

Intermediate Outcomes

illustration 1 pima community college pathways to healthcare program arizona
Illustration #1: Pima Community College Pathways to Healthcare Program (Arizona)
  • Basic elements
    • In partnership with county One-Stop, college provides training and credentials in 16 health care occupations, organized into 5 broad pathways (+ 3 broad levels), programs varying from 1-24 months. Targets low-income adults, substantially Latino population. Funded by federal HPOG program.
  • Signature strategies
    • Instruction: Upfront assessment and training planning, w/10-week contextualized basic skills development program if below required entry skill levels. Many programs offer streamlined/compressed formats, translate “clock-hours” at lower levels to prerequisites to higher levels.
    • Supports: More intensive and coordinated case management (via One-Stop) and academic advising and services (via college), financial support covers training and related expenses.
    • Employment: Outreach to engage local employers in training, internships, employment; use One-Stop services; MOUs linking to various county and state agencies.
illustration 2 year up national
Illustration #2: Year Up (National)
  • Basic elements
    • National non-profit operating in 8 urban areas. Partners with community colleges and major employers in financial services and information technology. Targets disadvantaged youth, 18-24, with high school diploma/equivalent.
  • Signature strategies
    • Instruction: Six-month “learning development” phase provides customized training in technical and contextualized basic and professional skills (e.g., communication, behavior). Earn 14+ college credits through agreement w/local colleges.
    • Supports: “High support, high feedback” environment—learning communities, pro-active counseling, structured group sessions, supportive services, and weekly performance-based stipend of up to $260 throughout the year. Post-program follow-up and engagement.
    • Employment: Second six-month internship phase places students in entry-level financial services/IT positions with major employers, who contribute about half of total program costs per participant.

Illustrative Evidence on More Explicit Career Pathways Approaches: Comprehensive, well-targeted “first step” training programs can have substantial impacts

Sectoraltraining (Maguire et al. 2010)

3 experienced CBOs provide customized short-term training in varied high-demand fields

Disadvantaged adults, HS+, careful screening (n=1,014)

Year Up (Roder & Elliott 2011)

National org provides 6 months customized training + 6 month paid internship in IT and finance sectors

Disadvantaged urban youth 18-24, HS+

Program articulates w/local colleges, completers earn 14+ credits

Results from small initial experiment (n=164)

examples of recent experimental findings on signature cp strategies
Examples of Recent Experimental Findings on Signature CP Strategies
  • Learning communities
    • As bridge program (Fein et al. 2006)
    • By linking courses (Sommo et al. 2012; Visher et al. 2012)
  • More intensive, specialized personal guidance and support
    • Coaching (Bettinger & Baker 2011)
    • Help with financial aid applications (Bettinger et al. 2012)
    • Student success courses + support (Weiss et al. 2011)
  • Increased financial support
    • Performance-based scholarships (Patel & Richburg-Hayes 2012)
    • Unconditional needs-based grants (Goldrick-Rab et al. 2011)
  • Psycho-social interventions
    • E.g., implicit theories of intelligence, normalizing worries about fitting in and success (Yeager & Walton 2011)
designing evaluations that matter
Designing Evaluations that Matter

Evaluations should be designed in ways that invite practitioners to make use of the results and adopt solid practices based on evidence. We need to recognize the role of motivated program leaders at the center of evaluation efforts, to ensure that these efforts advance program theory and practice, rather than merely fulfilling a funder request. Research tells us that organizations and their leaders need to own and trust information in order to use it. Partnership on the ground between practitioners and evaluators, along with the long-term support of committed public and private funders, is indispensable if the goal is to deliver evaluations that actually improve program quality and effectiveness.

    • Public/Private Ventures, Priorities for a New Decade: Making (More) Social Programs Work (Better), 2011.


  • Elucidate and address theories of program effectiveness
  • Develop strong common metrics that will be easily understood and widely accepted
  • Design to anticipate interest in scaling and replicating successful approaches
  • Foster appreciation for technical rigor (e.g., experimental designs)
  • Produce evaluation products useful to policymakers and practitioners
    • Emphasis on active interaction: evaluators-practitioner, practitioner-practitioner
example the isis evaluation
Example: The ISIS Evaluation
  • First national random assignment study of U.S. career pathways programs
  • Separate tests of nine relatively comprehensive career pathways programs, with implementation and cost-benefit studies
  • Central goals
    • Assess overall effectiveness of each program
    • Understand “why and under what conditions?”
      • Work within well-specified theory of change
      • Strong focus on impacts on exposure to program inputs, intervening outcomes, heterogeneity
      • Non-experimental analysis of mediators of impacts on primary outcomes
  • Strong emphasis on collaboration helps ensure findings will count…
collaboration increases chances that findings will matter
Collaboration Increases Chances that Findings Will Matter
  • Extensive stakeholder outreach + literature review identify career pathways focus in ISIS…
  • Early and ongoing exchanges foster interest and create positive climate for experiments among potential sites
    • Working w/sites as partners w/complementary expertise and interests
    • Building an ongoing “learning community”
      • Learn from experts on the team, peer-to-peer exchange, participation in external forums
      • Help plan and execute dissemination strategies
  • Maintain and extend relationships with wider stakeholder network
    • Including private and public funding agencies, policy makers, professional associates, advocates, practitioners,
    • Through meetings, website, public webinars, varied publications
    • On diverse interests: related projects, substantive topics/directions in ISIS, evaluation findings, evaluation products,
  • So that findings aren’t limited to a small number of reports, but shared in dynamic, diversified ways with engaged audiences
  • Career pathways framework useful in program and evaluation design, organizing emerging findings
  • Genuine partnerships with practitioners and other stakeholders key to designing evaluations that matter
  • Thank you!