Building Implementation Capacity to Improve Youth Outcomes

1. Building Implementation Capacity to Improve Youth Outcomes Allison Metz, Ph.D. Associate Director National Implementation Research Network University of North Carolina at Chapel Hill

2. Agenda • The Challenge • Implementation Science • Plan for Change • Active Implementation Frameworks

3. The Challenge: Recognizing the Gaps • Science to Service Gap • What is known to be effective is not what is selected to help students • Implementation Gap • What is selected is not used with fidelity and good outcomes • What is used with fidelity is not sustained for a useful period of time • What is used with fidelity is not used on a scale sufficient to broadly impact youth outcomes

4. Implementation Gap PRACTICE RESEARCH IMPLEMENTATION Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions.

5. PRACTICE RESEARCH IMPLEMENTATION Why Focus on Implementation? GAP “Children, youth, and families cannot benefit from interventions they do not experience.”

6. IMPLEMENTATION Effective NOT Effective from Mark Lipsey’s 2009 Meta-analytic overview of the primary factors that characterize effective juvenile offender interventions – “. . . in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.” Actual Benefits Inconsistent; Not Sustainable; Poor outcomes Effective INTERVENTION Unpredictable or poor outcomes; Poor outcomes; Sometimes harmful NOT Effective (Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

7. “Implementation science is the systematic study of variables and conditions that lead to full and effective use of evidence-based programs and other effective innovations in typical human service settings.” —Blase and Fixsen, 2010 National Implementation Research Network “Implementation Science” Download at: http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/

8. Best data show these methods, when used alone Do not Result in Implementation as Intended • Diffusion/ Dissemination of information • Training • Passing laws/ mandates/ regulations • Providing funding/ incentives • Organization change/ reorganization 5 to 10% return on investmentNECESSARY BUT NOT SUFFICIENT

9. Plan for Change: “Making It Happen” • Letting it happen • Recipients are accountable • Helping it happen • Recipients are accountable • Making it happen • Purposeful use of implementation practice and science • Implementation teams are accountable • —Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004); • Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)

10. To successfully implement and sustain evidence-based and evidence-informed youth violence prevention interventions, we need to know: • WHAT to doWhat is the intervention (e.g.identifiedprevention, intervention and enforcement strategies that are feasible and relevant)? • HOW to do itActive and effective implementation and sustainability frameworks (e.g. strategies to change and maintain behavior of adults) • WHO will do itOrganized, purposeful, & active implementation support from linked implementation teams

11. Q. How? • A. Effective Implementation • Changing the behavior of practitioners and administrators • Creating the setting conditions to facilitate these changes • Creating the processes to maintain and improve these changes in both setting conditions and behavior of well-intentioned adults • So that youth benefit

12. Active Implementation Frameworks ImplementationDrivers ImplementationStages Improvement Cycles Implementation Teams

13. Common features of successful supports to help make full and effective use of a wide variety of innovations Implementation Drivers

14. Positive Outcomes for Youth Effective Youth Violence Prevention Practices Core Implementation Components Staff capacity to support youth with the selected practices Institutional capacity to support practitioners in implementing practices with fidelity Organization Drivers Competency Drivers Leadership Capacity to provide direction and vision © Fixsen & Blase, 2008

15. Improved youth outcomes Consistent Use of Prevention Innovations Interventions meet Implementation Performance Assessment (Fidelity) Systems Intervention Coaching Facilitative Administration Adaptive Training Competency Drivers Organization Drivers Integrated & Compensatory DecisionSupport DataSystem Selection Technical Leadership © Fixsen & Blase, 2008

16. Purposeful matching of critical implementation activities to the stage of the process Implementation Stages

17. Stages AND Drivers Implementation Takes Time: 2 – 4 Years EXPLORATION INSTALLATION Drivers INITIAL IMPLEMENTATION Drivers FULL IMPLEMENTATION Drivers “DRIVERS”

18. Stages of Implementation “Pay now or Pay later” EXPLORATION CompetencyDrivers OrganizationDrivers Integrated & Compensatory Leadership Drivers —Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

19. Goals of Exploration • Create readiness for change • Changing hearts and minds • Examine degree to which the proposed strategies and practices meet the needs of our community and our youth • Determine whether the strategies, practices, and implementation are desirable and feasible

20. The HexagonAn EBP Exploration Tool • Need in school, district, state • Academic & socially significant Issues • Parent & community perceptions of need • Data indicating need The “Hexagon” can be used as a planning tool to evaluate evidence-based programs and practices during the Exploration Stage of Implementation. Download available at: www.scalingup.org/tools-and-resources • Capacity to Implement • Staff meet minimum qualifications • Able to sustain Imp Drivers • Financially • Structurally • Buy-in process operationalized • Practitioners • Families NEED • Fit with current Initiatives • School, district , state priorities • Organizational structures Community values CAPACITY FIT READINESS RESOURCES • Resources and supports for: • Curricula & Classroom • Technology supports (IT dept.) • Staffing • Training • Data Systems • Coaching & Supervision • Administration & system • Readiness for Replication • Qualified purveyor • Expert or TA available • Mature sites to observe • Several replications • How well is it operationalized? • Are Imp Drivers operationalized? EVIDENCE • Evidence • Outcomes – Is it worth it? • Fidelity data • Cost – effectiveness data • Number of studies • Population similarities • Diverse cultural groups • Efficacy or Effectiveness © National Implementation Research Network 2009-2012 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland

21. Stages of Implementation “If you build it, they will come”. . . but you actually have to build it! INSTALLATION CompetencyDrivers OrganizationDrivers Integrated & Compensatory Leadership Drivers —Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

22. Goals of Installation • Structural and functional changes are made to support implementation • Staff selection protocols developed • First ‘practitioners’ selected • Define and initiate training of first cohort of practitioners • Develop coaching system and plans • Evaluate readiness and sustainability of data systems (e.g. fidelity, outcomes)

23. Stages of Implementation “Get Started, then Get Better.” INITIAL IMPLEMENTATION INSTALLATION CompetencyDrivers OrganizationDrivers Integrated & Compensatory EXPLORATION Leadership Drivers Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

24. Goals of Initial Implementation • Work through the Awkwardness • Provide training and coaching on the evidence-based practice, re-organization of school roles, functions and structures • Make use of improvement cycles to resolve systems issues • Continue buy-in efforts and manage expectations • All the components of the program or innovation are at least partially in place and the implementation supports begin to function

25. Stages of Implementation FULL IMPLEMENTATION 2 - 4 Years “The only thing worse than failing and not knowing why you failed, is succeeding and not knowing why you succeeded.” Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

26. Goals of Full Implementation • Maintaining and improving skills and activities throughout the system • Components integrated, fully functioning • Skillful practices by front line staff, supervisors, administrators (50% meet performance criteria) • Changes in policy are reflected in practice at all levels • Ready to be evaluated for expected outcomes

27. IMPROVEMENT CYCLES Changing on purpose to support the new way of work

28. Improvement Cycles PDSA Cycles Act Plan Study Do

29. New practices do not fare well in existing organizational structures and systems • Effective innovations are changed to fit the system, as opposed to existing systems changing to support effective innovations. • People, organizations, and systems. . . • Cannot change everything at once (too big; too complex; too many of them and too few of us) • Cannot stop and re-tool (have to create the new in the midst of the existing) • Cannot know what to do at every step (we will know it when we get there) • Many outcomes are not predictable (who knew!?)

30. Plan-Do-Study-Act Cycles • Rapid cycle problem solving (Shewhart; Deming) • Transformation Zone • Usability testing (Neilson; Rubin) • Practice-policy communication loops

31. External Implementation Support Practice-Policy Communication Cycle Policy Policy Practice Informs Policy Policy Enables Practices Plan Structure Feedback Study - Act Procedure Practice Do Practice FORM SUPPORTS FUNCTION

32. IMPLEMENTATION TEAMS Organized, expert assistance to develop and sustain an accountable and effective structure

33. Implementation Teams • Provide accountable and effective structure to move intervention through stages of implementation • Scope of the initiative determines the number of teams and the linked communication protocols needed • Focus is on • Ongoing “buy-in” and readiness • Installing and sustaining the Implementation Drivers • Fidelity & Outcomes • Systems Alignment and Stage-based work • Problem-solving and sustainability

34. Implementation Teams Linked Team Structures State-based Implementation Team “We tend to focus on snapshots of isolated parts of the system and wonder why our deepest problems never seem to get solved.” —Senge, 1990 Regionally-based Implementation Team Community-based Implementation Team Site-based Implementation Team

35. Implementation Teams Who Makes Change Happen? SO THAT... …can effect change throughout the system and make decisions …can promote implementation of core components, adaptations, and infrastructure …can guide implementation and build capacity throughout the organization and system …can develop and follow-through on action planning …can support efforts to improve access, reach or scale, improve connection, influence decision-making

36. Examples of Success Implementation results (higher fidelity) and intervention results (improved outcomes) improve when Active Implementation Frameworks are used purposefully and systematically • Catawba County Child Wellbeing Project- 5 year period to develop, implement and evaluate post-care service system for children exiting foster care; high fidelity and early indicators of success • Colorado EPIC Project- Scale-up of MI using Active Implementation Frameworks in corrections • EPIS Center at Penn State University - 419 age-grade cohorts over a 5-year period: youth in CTC communities using EBPs had significantly lower rates of delinquency, greater resistance to negative peer influence, stronger school engagement and better academic achievement • Evidence-Based Model Purveyors – NFP, MST, FFT use many components of active implementation frameworks to achieve sustainable outcomes

37. Summary: “Making it Happen” for youth • Purposeful selection of an effective and feasible “What” • Conceptualize a change process so that effective interventions for children and families can become embedded and sustained in socially complex settings • “stage-matched activities” to guide the process • “implementation drivers” to build the infrastructure • Improvement processes are critical • the work is never done because the environment is in motion • Invest in the development of organized, “expert” implementation support

38. Stay Connected! Allison.metz@unc.edu nirn@unc.edu nirn.fpg.unc.edu www.scalingup.org www.implementationconference.org