EBP in the Real World

1. EBP in the Real World A Provider’s Perspective on the Use of Evidence-Based Programs and Practices

2. Operation PAR, Inc. • Mission: To strengthen our communities by caring for families and individuals impacted by substance abuse and mental illness. • Vision: Operation PAR, a beacon of hope in our communities—helping people be aware, be responsible, be healthy and be happy.

3. Learning Objectives • To encourage audience discussion of “Evidence-Based Programs and Practices” including a review of the literature and practical definitions and applications. • To discuss the importance of training and certification in Evidence-Based Programs and Practices to ensure fidelity and adherence to the model. • To share Operation PAR’s experience implementing SAMHSA/CSAT projects involving the use of two evidence based practices (A-CRA/ACC and the GAIN) over a 5 year time span.

4. The Research Perspective What constitutes an evidence-based practice Why it matters

5. Defining “Evidence-based practice” • Core elements: • Scientifically tested • Results are replicable • Clearly defined training protocol • Clearly defined adherence protocol • Measurable outcomes*

6. Key Definitions • Diffusion: The “grape vine.” • Dissemination: Methods by which therapist competency or skillfulness in a particular evidence-based approach are taught and strengthened. • Implementation: Agency-level methods for adopting and integrating an evidence-based practice into clinical programs.

7. Issues in translating research to practice • Why does a gap exist between research and the “real world”? • The impact of the “grape-vine” effect • What we don’t know we don’t know • Barriers to individual clinician buy-in • The “cookie-cutter” or “cookbook” effect • Organizational or systemic barriers • Changing business as usual

8. The Great Divide: Why a gap exists between research and the real world The Great Divide: Why a gap exists between research and the real world

9. The Disconnect: Researchers versus Clinicians

10. Individual Clinician Resistance: Fact or Fiction?

11. The Agency

12. Key Findings in Literature • Traditional dissemination methods (e.g., manuals, workshops) are ineffective in building competencies and instilling intrinsic motivations to adopt evidence-based approaches, yet they are the most common. • Over the last decade, the field has been literally flooded with “evidence-based” practices, BUT: • Qualifiers for what constitutes “evidence-based” varies • Valuable information on potentially beneficial EBP’s flows through the wrong channels • Clinicians and upper management are too busy with day-to-day operations to keep up

13. Key Findings Continued… • We are witnessing a shift in the methods by which researchers and policy-makers determine which treatment should be translated into practice. • Treatment Improvement Protocols have been widely used and are reflective of a consensus of experts in the field • Others blend research findings with expert opinion to offer guidelines • More recently, there has been an increasing focus on summary reviews and meta-analyses of published research, often with the goal of producing a listing of treatments which could be defined as “evidence-based”

14. NREPP • National Registry of Evidence-based Programs and Practices • Dates back to the 1990’s, when an effort was launched to collect information on effective treatment interventions. • A report by SAMHSA Science to Service Coordinator, Kevin Hennessey noted (see reference below) that by 2003: • 1,100 interventions had been submitted for NREPP review • Of these, 150 were listed on NREPP • NREPP initially targeted prevention projects, since it was born under a CSAP initiative. Source http://www.jbassoc.com/reports/documents/panels/panels/panel%208/panel8hennessy.pdf

15. NREPP • By 2003, SAMHSA realized the need to expand NREPP reviews to include: • Substance abuse interventions • Mental health interventions • Interventions which treat co-occurring disorders • In response to a solicitation for input, the public asked for NREPP changes to: • Provide summaries that were easy to understand and were “transparent.” • Prohibited agencies from mandating NREPP interventions Source http://www.jbassoc.com/reports/documents/panels/panels/panel%208/panel8hennessy.pdf

16. NREPP helps by providing: A publicly available list of interventions reviewed by NREPP Results of independent reviews which are posted for consideration Strict, standardized criteria used across all types of interventions to ensure uniformity

17. Source: http://www.nrepp.samhsa.gov/AboutNREPP.aspx

18. How can this benefit…

19. NREPP Review Process

20. NREPP Review Process Both Quality of Research and Readiness for Dissemination are rated on a scale of 1.0 to 4.0

21. NOTE: Though one of the program examples utilized Motivational Enhancement Therapy in conjunction with Cognitive Behavioral Therapy, CBT has not formally undergone the NREPP review process and thus is excluded from the following slides NREPP Ratings • Let’s compare the NREPP ratings for the EBP’s utilized in the Operation PAR programs under discussion today: • Motivational Enhancement Therapy (MET) • Family Support Network (FSN) • Adolescent Community Reinforcement Approach (A-CRA)

22. How is Quality of Research Rated? • The dimensions of “Quality of Research” discussed previously (e.g., Reliability of measure, Validity of measure, etc.) are rated according to the outcomes reported in the published research provided as evidence of the program’s effectiveness. • This will vary across interventions.

23. MET: Quality of Research Table 1. Quality of Research Criteria for Motivational Enhancement Therapy (MET)

24. FSN: Quality of Research Table 2. Quality of Research Criteria for Family Support Network (FSN)

25. A-CRA: Quality of Research (part 1) Table 3. Quality of Research Criteria for A-CRA

26. A-CRA: Quality of Research (part 2) Table 4. Quality of Research Criteria for A-CRA

27. Cross Comparison: Quality of Research

28. Readiness for Dissemination Table 5. Comparison of Readiness for Dissemination Ratings

29. The Clinical Perspective The importance of training and certification Why fidelity matters

30. Each EBP includes a specific training and certification process Evidence-based practices included: • Global Appraisal of Individual Needs (GAIN) • Adolescent Community Reinforcement Approach • Assertive Continuing Care • Motivational Enhancement Therapy/Cognitive Behavioral Therapy • Family Support Network

31. GAIN Training and Certification • Different types of certification • GAIN Administration • GAIN Clinical Interpretation • Different levels of certification • Site Interviewer • Local Trainer • Site Interpreter • GAIN Clinical Local Trainer Source http://www.chestnut.org/LI/gain/GAIN%20Training/index.html

32. A-CRA/ACC Training and Certification • Different types of certification • A-CRA • ACC • Different levels of certification for both A-CRA and ACC • Clinician certification • Supervisor certification Source http://www.chestnut.org/LI/acra-acc/index.html

33. MET/CBT & FSN Training and Certification • Different types of certification • Different levels of certification • Clinician certification • Supervisor certification Source http://www.chestnut.org/LI/cyt/products/index.html#treatment

34. Common Elements of Training • Structured training manual • Standardized training process • Training includes variety of teaching methods: • Lecture style • Demonstration • Hands on practice

35. Common Elements of Certification • Requirement to audio or video record sessions (treatment or assessment/interview) • Submission of audio recordings for quality assurance review • Each EBP has a corresponding standardized procedure for evaluating the quality of sessions or interviews • Written feedback is provided to candidate • Process repeats until certification is achieved • Certification Deadlines

36. How does this really work?

37. Supervisor Complaints: • Time investment for staff • Handling staff complaints • Doubts efficacy of “new” model • Program-level change is complex • Often insufficient/non-existing funding for new intervention • Clinician Complaints: • Perceived threat to competency • Not engaging in certification • Disdain for “cookbook” therapy • Feel clinical skills are lost • Issues with audio recordings Challenges

38. Prochaska & DiClemente’s Stages of Change Model

39. Rogers’ (2003) application of Stages of Change to individual clinicians

40. Benefits • Supervisors: • Supervision simplified yet thorough • Positive program outcomes • Increased attendance • Increased engagement • Clinicians: • Clinicians are more marketable • Achieve certification • Become an “expert” in the model • Learning the model enables flexibility in clinical style

41. Speaking a Common Language • Researchers and Clinicians • Understanding what we’re both talking about • Clinicians and Clinicians • Differences in clinical studies and clinical programs

42. Reconciling Paperwork Requirements • Differences in clinical charting • Agency requirements • A-CRA requirements

43. The Program Manager’s Perspective How it works in the real world Challenges, successes, and hopes for the future

44. Bay Area Young Offender Reentry Program (YORP) #1 • CSAT Grant (TI16928) funded from 2005 through 2009 • Targeted sentenced, substance abusing, adolescents (age 13 to 18) who were within 60 days of being released from a Florida Department of Juvenile Justice (DJJ) residential commitment program. • EBP’s utilized: • Global Appraisal of Individual Needs (GAIN) • Motivational Enhancement Therapy/Cognitive Behavioral Therapy - 12 sessions (MET/CBT 12) • Family Support Network (FSN)

45. YORP #1 Implementation Challenges • Extreme delays in IRB approval resulted in extreme delays in recruitment/enrollment • IRB approval was received, however: • Removed our permission to recruit youth while in commitment • Reduced referral sources by half • Staff turnover • Clinicians had virtually no buy-in to EBP’s

46. Impact on Treatment Engagement

47. Impact on Treatment Outcomes

48. YORP #1 Lessons Learned • Start IRB review process as early as possible • Contact IRB for review guidelines or requirements prior to submission, if possible • Have a back-up plan for additional referral sources • Staff clinicians who can embrace and fully adopt the EBP • Schedule regular meetings • Use data regularly to inform program adaptations • The data and corresponding adaptations must be timely to be effective

49. PAR Adolescent Recovery Intervention Services (PARIS) #1 • CSAT Grant (TI17761) funded from 2006 through 2009 • Targeted substance abusing adolescents (age 13 to 17) who were referred for substance abuse treatment from a variety of agencies (school, probation, community, etc.). • EBP’s utilized: • Global Appraisal of Individual Needs (GAIN) • Adolescent Community Reinforcement Approach (A-CRA) • Assertive Continuing Care (ACC)

50. PARIS #1 Implementation Challenges • First research grant and program at Operation PAR to utilize A-CRA/ACC • Clinical staff turnover • 1 was a poor fit and 1 was fired for performance issues • Issues with clinical buy-in were still present, but not nearly as pronounced as seen in YORP #1 • The treatment was provided in-home, which was a shift for most clinicians who started on the grant • The treatment site was located farther off site than any program before • Site was a 40 mile round trip drive