Factors Contributing to Successful Implementation of EBPs

2. Factors Contributing to Successful Implementation of EBPs • The success of the intervention strategies themselves and • The success of the implementation processes Too often, evaluations focus more on changes in client outcomes without due consideration of fidelity to the intervention model or the effectiveness of the implementation process itself.

3. Implementation Defined • Implementation is a specified set of activities designed to put into practice an activity or program of known dimensions. • The National Implementation Research Network at the University of Southern Florida conducted an exhaustive review of research on implementation and is the primary source for this presentation.

4. A Conceptual Framework for Understanding Implementation of Evidence-Based Practices The essential implementation outcomes are: • Changes in adult professional behavior • Changes in organizational structures and cultures • Changes in relationships to consumers, stakeholders, and system partners

5. Implementation Framework

6. Stages of Implementation • Exploration & Adoption • Program Installation • Initial Implementation • Full Operation • Innovation • Sustainability

7. Stages of Implementation Exploration & Adoption • Identify the need for an intervention considering existing conditions • Assess the fit between the intervention and program and community needs • Prepare organization, staff, and resources for mobilizing information and support. This was essentially accomplished in the work of the Work Groups.

8. Program Installation Program installation begins after an adoption has occurred and involves: • Ensuring availability of funding • Human resource strategies (recruitment, hiring, training, etc.) • Policy development to support change (e.g. referral mechanisms, reporting frameworks and outcome expectations) • Facility requirements

9. Initial Implementation • Initial implementation begins once conditions identified in the installation phase have been sufficiently completed. • Initial implementation must NOT be confused with full operation, as during initial implementation, many factors contributing to fidelity to the model may not be fully or even partially in place.

10. Barriers to Full Operation Organizational Barriers • Personnel rules • Social stressors • Union stewards • Anxious administrators • Political pressures • Interpersonal rivalries • Staff turnover

11. Barriers to Full Operation Human Barriers • Fear of change • Inertia • Investment in the status quo • Inherently difficult work of doing something differently

12. Full Operation Full Operation ONLY occurs when: • New learning becomes integrated into practitioner, organizational and community practices, policies, and procedures. • Full case loads • Full staffing • Fidelity to the model being implemented Only once fidelity measures are above criterion levels most of the time, can the effectiveness of an evidence-based practice be evaluated. This is a very important point.

13. Dangers of Premature Evaluation “ Outcome evaluations should not be attempted until well after quality and participation have been maximized and documented in a process evaluation. Although outcome data can determine the effectiveness of a program, process data determine if a program exists at all. Gilliam, Ripple, Zigler & Leiter (2000). ”

14. Innovation or Drift When evidence-based practice meets new local conditions. Two results can occur: • Innovation: When desirable changes in the standard model are identified. • Drift: Undesirable changes in the standard model. It is critical to implement the model with fidelity before considering innovation and to make changes only after due consideration of client benefit.

15. Innovation or Drift Illustrated

16. Sustainability Throughout the two to four-year implementation process, the forces that led to and supported reform will change: • Staff and funding changes • New social problems emerge • Partnerships and political alliances change • Champions and advocates move on to other causes Throughout the implementation process, it is essential that leadership maintain focus on sustaining the core components of reform.

17. Factors Contributing to Effective Implementation of EBP • Objective decision-making strategies that involved staff, good information about the reform and organizational leadership support during the exploration stage; • Evidence of a ‘learning culture’ within the organization implementing the reform; • A system in place for monitoring implementation; • Access to technical assistance throughout implementation; • The perceived ability of the organization to manage risks; • Belief in the validity of the reform; and most of all…. • Adherence to the core components of BOTH the EBP and effective implementation (discussion follows).

18. Core Intervention Components An essential first step to implementing EBP or system reform is to identify what are the core components of the intervention itself. Leadership must: • Carefully research alternatives • Consider well-evaluated experiential learning from a number of replications and • Achieve a clear understanding of what of the model must be maintained to achieve fidelity and effectiveness at the consumer level. In other words, before considering the core components of any intervention, leadership must be clear about what is to be implemented.

19. Core Components for Implementation The following Implementation Drivers influence staff behavior and organizational culture: • Practitioner Selection • Preservice and inservice training • A consultant coach • Staff & program evaluation • Facilitative administration and • System Interventions This is illustrated in the graphic that follows.

20. Implementation Drivers

21. Implementation Drivers These drivers are integrated in their influence, in that strengths in one area can compensate for weaknesses in other areas. The relative effectiveness of the implementation drivers is as important as the demonstrated effectiveness of the EBP itself.

22. Practitioner Selection How and who are selected to carry out the EBP • Who is qualified to carry out the practices and programs? • What are the methods for recruiting & selecting practitioners? • How important are individuals skills, experience and personal characteristics to effective implementation of EBP?

23. Research on Practitioner Selection There has been relatively little research to isolate the factors involved in selection of practitioners for EBP. Factors identified in effective practitioner selection for a national MST program included: • Responses to behavioral vignettes; • Responses to role playing situations related to the therapeutic; environment for which they were being considered; and • Responses to mini-training that requires behavior change. Responses are rated with a rubric along several dimensions including: collaborative and strength focused, efforts to overcome barriers, ability to use behavioral language, uses of logical thinking and openness to feedback.

24. Research on Practitioner Selection Morris & Stuart, 2002 are attempting to identify the generic skills needed by frontline practitioners in a transformed behavioral health field. Among those qualities considered: • Assessment skills; • Family and support system involvement; • Social and cultural engagement skills; • Treatment skills; • Methods to optimize recovery and empowerment; • Consumer relationship skills; and • Community resource and coordination skills. It is interesting to note here and throughout this research, cultural competence was never discussed as an issue or factor.

25. Research on Staff Selection A Housing & Urban Development study by Wanberg & Banas (2000) examined practitioner characteristics in the context of organizational change and found that personal resilience and self-efficacy were associated with greater acceptance of change in the work place. This study would seem particularly important in SMCMH’s system transformation.

26. Research on Staff Selection From the research we might conclude that as part of the implementation planning process, SMCMH leadership should: • Construct a rubric of ideal staff characteristics particularly as relates to operating in a transformed organization • Design an interview process that includes vignettes and role plays that require candidates to demonstratethose characteristics

27. Preservice and Inservice Training Training is an efficient way to provide: • Background knowledge, background information, theory, philosophy, and values; • Demonstration of new skills (through video or role play); • Opportunities to practice new skills through role plays and behavioral rehearsals; and • Feedback in a safe training environment.

28. Preservice and inservice training Role playing & behavior rehearsals are critical for practicing new skills in training. • Role playing asks you to pretend you are someone else and try this… which builds empathy • Behavioral rehearsals asks you to be in your practitioner role and you are asked to confront a specific situation and perform your practitioner role in reaction to that situation ….which serves as direct preparation for the real thing.

29. Recommendations for Training • Model or demonstrate new skills using role play, behavior rehearsals and video tape. • Emphasize practice and use feedback on practice to teach the finer points of mapping. • Use practice sessions to help trainees integrate thinking and doing. • Provide guidance with respect to the boundaries of using the technique, describing when it may be useful and when it may not be useful. • Provide guidance on the flexible use of the core components. • Encourage peer and administrative support.

30. Limitations of Training Numerous evaluations on the impact of training in health and human services have found that training alone rarely impacts practice. • The “train-and-hope” approach (Stokes & Baer, 1977) to implementation does not appear to work. • Kelly et al (2000) in a study of HIV service organizations reported the largest increase in adoptions of HIV service guidelines occurred when consultation was added to training. • A meta-analysis (Davis, 1995) found similar results in medicine. Davis concluded that “formal CME conferences and activities without enabling or practice reinforcing strategies, had little impact.” (page 700)

31. Limitations of Training: Spray & Pray While training may introduce knowledge, philosophy, and new approaches, there is no research study that has demonstrated significant changes in practice resulting from an intervention that provided only training. The limitations of training are directly related to the way in which adult learning occurs.

32. Stages of Adult Learning Generally adult learning progresses in stages: • Orientation & new learning • Mechanical use • Routine use • Refinement • Integration • Innovation

33. Challenges of Learning New Behaviors Based upon decades of research, Joyce & Showers (2002) concluded that • The newly-learned behavior is crude (i.e. mechanical) compared to performance by a master practitioner. • Newly-learned behavior is fragile and needs to be supported in the face of reactions from consumers and others in the service setting. • Newly-learned behavior is incomplete and will need to be shaped to be most functional in a service setting.

34. The Importance of Coaching While most skills can be introduced in training, newly-learned behaviors are only really learned on the job with help of a consultant or coach precisely because: • The challenges of adopting new behaviors can’t be replicated in a training. • Single-point-in-time training needs to be reinforced continuously in the work setting for behavior changes to be sustained.

35. Consultant or Coach Support • Implementation of EBP requires changes in behavior at the practitioner, supervisory and administrative levels. • Training and coaching are the primary strategies in which behavior change is brought about. In addition to training, substantial hands-on coaching and practice may be necessary before a counselor feels comfortable with a new strategy. ~ Dansereau & Dess (2002) “ ”

36. Components of Effective Coaching Coaching must be work-based, opportunistic, readily available, and reflective (Spouse 2001). Spouse described four roles for the coach: • Supervision • Teaching while engaged in practice activities • Assessment and feedback • Provision of emotional support.

37. Factors that Impact Coaching • Amount of time devoted to coaching (MST coaching occurs once or twice a week for 90 minutes) (Schoenwald et al., 2000) • Includes direct observation of provision of direct services (Smart et al., 1979) • Utilizes coaches who are expert in the content, techniques, and rationales of the program (Denton, Vaughn & Fletcher 2003) • Coaching relationships established during the training experience (Smart et al., 1979)

38. Other Factors Contributing to Effective Coaching Walker, Koroloff & Schutte (2002) identified four additional factors that accounted for 62% of the variance in the perceived impact of supervision and coaching on practice. The Supervisor (coach): • Taught new skills; • Strengthened confidence; • Offered safety in sessions; • Devoted time to discipline-specific skills

39. Personal Qualities of a Coach • Encouraging & enthusiastic • Supportive • Committed • Sensitive • Flexible • Respectful & diplomatic • Willing to share information, credit and recognition.

40. Barriers to Effective Coaching • Inadequate time allotted to coaching & scheduling conflicts • Role confusion due to the dual role of supervisor and coach • Feelings of inadequacy on the part of coaches (hence the importance of selecting skilled coaches) • Poor match between coach and practitioner • Labor relationships that don’t support observation and feedback • Resistance in the organizational culture • Absence of strong leadership & commitment to implementation • Focus on paperwork compliance as opposed to changes in practitioner behaviors • Cost

41. Research on the Impact of Coaching The study below is based on an meta-analysis of hundreds of studies in education. Note the dramatic differences in impact upon practice.

42. Staff Evaluation Evaluation provides critical feedback to practitioners, trainers and managers related to: • Fidelity to the model being implemented • Effectiveness of training and coaching strategies • Impact of intervention upon clients • Progress of implementation itself

43. Components of Evaluation Staff & program evaluation and fidelity seem to consist of some combination of measures of: • Context—prerequisites that must be in place to operate (staffing, qualifications, ratios) • Compliance—extent to which the practitioner uses the core intervention components as prescribed by the EBP and avoids practices proscribed by the EBP • Competence– the level of skill shown by the practitioner in using the core intervention components

44. Effective Staff Evaluation Systems Huber et al., (2003) described a highly effective hospital management system for staff recruitment and evaluation that included: • Ongoing training and education focusing on specific skills; • Cross-training on related roles, and in-services • Monthly dinners for discussion; • Performance evaluations based on direct observation to assess practice knowledge, communication skills, and use of time; • Prompt verbal feedback followed by a write up with recommendations; and • Quality improvement systems to keep the system on track.

45. Effective Staff Evaluation In other words, in a highly effective system, staff evaluation is part of a sequence of supports designed to have good people well prepared to do an effective job.

46. Staff Evaluation & Fidelity to a Model One of the most critical purposes of staff evaluation in the context of implementing EBP is to test practitioner fidelity to the EBP core components.

47. Challenges to Measuring Fidelity McGrew et al., (1994) noted that development of fidelity measures is hampered by three factors: • Most treatment models are not well defined conceptually making it difficult to identify core intervention components; • When intervention components are identified, they are not operationally defined with agreed-upon criteria; and • Only a few models have been around long enough to study planned and unplanned variations.

48. Staff Evaluation & Fidelity to a Model Fortunately the SMCMH transformation relies upon implementation of EBPs with effective fidelity instruments already in place. • Multisystemic Treatment (MST) utilizes the Therapist Adherence Measure, a 27-item measure used in phone interviews with parents. • The Wraparound Fidelity Index (WFI) consists of asking facilitators, parents and youth to rate 11 dimensions of services. • Assertive Community Treatment (ACT) utilizes a 73-item tool with 17-item subset used to construct a fidelity index with three subscales: staffing, organization, and service.

49. Fidelity to System Transformation A major challenge to SMCMH is to go beyond program level (MST, Wraparound, ACT) fidelity and to construct a fidelity index that reflects the qualities of a transformed system as defined and described in our plan.

50. Theory of Change & Fidelity Models Possible resources to explore for developing a fidelity model for the transformation… • Hodges, Hernandez, Nesman & Lipien (2002) demonstrated how a theory of change exercise can help programs clarify their strategies to develop fidelity measures. • Shern, Trochim & LaComb (1995) used concept mapping to develop fidelity measures for a mental health program.