® S-START Evaluation Evaluation Team Evaluator: Nancy Amodei, Ph.D. – Dept Pediatrics Evaluation Coordinator: Danielle Dunlap, M.S. – Dept Pediatrics Data Manager: Kyle Kozlovsky, M.S. – Dept Pediatrics Qualitative Expert: Suyen Schneegans, M.A. – Dept Pediatrics Special thanks to: Rasheem Battle Alejandro Bocanegra Meghan Crabtree Merced Doria Destiny Ramos Drew Russell
Process Evaluation: How is the program being implemented? • S-START Process Goals • Train UTHSCSA medical residents and residents from other participating South Texas programs to use evidence based SBIRT procedures for patients who have or are at risk of substance abuse disorders. • Promote systems change in targeted residency programs by integrating the SBIRT model into the curriculum on a long-term basis.
Process Goal #1: Train UTHSCSA medical residents and residents from other participating South Texas programs • Key Activities and Measures 1) Develop/implement a comprehensive curriculum 2) Train UTHSCSA and other Faculty • Demographics, type of training, satisfaction (GRPA & qualitative findings) 3)Train UTHSCSA residents & residents from other programs • Demographics, type of training, satisfaction (GPRA)
Baseline vs. 30-day Faculty GPRA ratings (N=17) * Wilcoxen Signed Ranks Test * Wilcoxon Signed Ranks Test
Qualitative Study of Faculty Perceptions of S-START • Purpose: Gain an in-depth understanding of the experience and perceptions of S-START faculty • Methods • 16 training faculty from 5 specialties invited • 15 accepted (12 from UTHSCSA; 1 FM program in McAllen, 1 FM CHRISTUS Santa Rosa, 1 FM from Fort Hood) • Mean age (43.93 years); 72% female; 73% MDs, 1 Ph.D., 1 PsyD, 1 M.A. • Average yrs of experience = 15
Qualitative Study of Faculty Perceptions of S-START • Data Collection • Data collection: ≈ 22 months after S-START began • 45 to 60 minute interviews using scripted but open-ended questions • 14 of the interviews taped to facilitate transcription • Topics: How S-START implemented in the program, barriers and challenges, impact of potential clinical service reimbursement in facilitating program; suggestions for improvement
Qualitative Study of Faculty Perceptions of S-START • Data Analysis • Evaluation team hand-coded transcribed interviews • Thematically coded them to correspond to each question • Collapsed materials thematically into 10 emergent or preset categories
Qualitative Study of Faculty Perceptions of S-START • Results 3 Thematic categories accounted for > 50% of interview responses • Critical components • Barriers • Motivation Critical Components • Faculty training Barriers • Lack of Leadership Motivation • Buy-in from faculty and residents
Baseline vs. 30-day Resident GPRA ratings (N=409) * Wilcoxon Signed Ranks Test
Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SBIRT model into curriculum on long-term basis • Key Activities • Council of Residency SBIRT Trainers Meetings • Elicit support of key personnel • Changes to Electronic Medical Record • Pocket Cards • SBIRT resources (including key modules) on the S-START website • iPad Project
Process Goal #2: Promote Systems Change in Targeted Residency Programsby integrating SPIRT model into curriculum on long-term basis Progress re other Activities • Support of change leaders- e.g. UTHSCSA President, Residency Program Directors; PD and Co-PD have high profile positions • Changes to Electronic Medical Record- • UTHSCSA – DFCM, Peds; (Psychiatry and Surgery planned) • Pocket cards • McAllen FM: Part of every patient visit paperwork • SBIRT resources (including core modules, resource directory) on the S-START website • iPad Project-proposed for UTHSCSA Pediatrics
What is the program’s impact? S-START Outcome Goals: • Enhance residents’ knowledge of evidence-based SBIRT practices. • Enhance residents’ readiness and perceived confidence to implement SBIRT with their patients • Increase residents’ implementation of SBIRT practices with their patients • Enhance Faculty Participants’ knowledge and confidence in ability to teach SBIRT practices to future physicians
Outcome Design • 3 x 2 Repeated Measures • Three data collection methods • Measurement Occasions for Surveys: • Pre-Test • 30-Day Follow-Up • Annually up to 36-month follow-up • Measurement Occasions for Pocket Cards • Varies by department • Measurement Occasions for Chart Reviews • 12-month period prior to first core SBIRT module implementation • 12-month period following the first year of SBIRT module implementation • 12-month period following the third year of SBIRT module implementation
Methods of Survey Data Collection • Web-based surveys (i.e., SurveyMonkey) • Emails to UTHSCSA and private email addresses • Unique web links provided to residency coordinators • Hard copy surveys • Pass out at grand rounds and conference periods • Intra-office mail for fellows, faculty • Mail to home and clinic physical address
Strategies for survey follow-up • Collected contact information using a comprehensive tracking form • Text reminders to cell phone numbers • Phone calls to (1) cell, (2) home, (3) significant others, (4) clinic • Contact residency coordinators for updated contact information • Enlist authoritative support of faculty • Look up information using White Pages, AMA DoctorFinder, respective state medical board websites (usually Texas) • Peer-to-peer contact updates
Future strategies for survey follow-up • Reminder postcards sent twice before each annual survey • Bring surveys to end-of-year gatherings for graduating residents • Include surveys in residents’ exit processing before graduation I pity the fool who doesn’t take the survey.
Resident survey rates • Response rates for similar populations (e.g., students, medical professionals) tend be 60% or lower on follow-up surveys (Asch et al., 1997; Kaplowitz et al., 2004; Kaspryzyk et al., 2001; McMahon et al., 2003; Porter & Whitcomb, 2007)
Analyses of resident survey data • Demographic data (pre-test) • Measured changes from pre-test to 12-month follow-up in: • Confidence to use SBIRT (residents only) • Readiness to use SBIRT (residents only) • Current SBIRT practice (residents only) • SBIRT knowledge (residents & faculty) • Confidence to teach SBIRT (faculty only) • Selected departments for analysis: Pediatrics, Family and Community Medicine, Internal Medicine
Resident DemographicsPre-test results Note. UT=University of Texas Health Science Center at San Antonio; Ped.=Pediatrics; FM=Family Medicine; IM=Internal Medicine; OB=Obstetrics/Gynecology; Psy.=Psychiatry (adult & child); Sur=Surgery. aSurgery residents began the SBIRT curriculum on August 15, 2011.
Resident Demographics (cont.)Pre-test results Note. UT=University of Texas Health Science Center at San Antonio; FCM=Family and Community Medicine; McA.=McAllen Family Medicine; SR=CHRISTUS Santa Rosa Family Medicine; FH=Fort Hood Family Medicine; IM=Internal Medicine; ERAHC=Edinburgh Regional Academic Health Center Internal Medicine.
Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices. • Core SBIRT knowledge • 12 items developed locally by the SBIRT project directors • Knowledge that residents across all departments should know after training • Residency-specific SBIRT knowledge • 7-17 items developed locally by the SBIRT faculty in the respective programs • Items designed for specific residency program SBIRT knowledge and patient populations
Outcome goal 1Enhance residents’ knowledge of evidence-based SBIRT practices. • Sample core knowledge item: • “How many ‘standard drinks’ are considered at-risk alcohol use by a healthy 40-year-old man?” • Sample Pediatrics knowledge item: • “________ exposure is the leading known preventable cause of mental retardation.” • Sample Family and Community Medicine knowledge item: • “Hepatitis B, hepatitis C, HIV and AIDS are strongly associated with abuse of…” • Sample Internal Medicine item: • “Alcohol withdrawal treatment on the inpatient medical service is best accomplished by…”
Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices. • Core SBIRT knowledge • All residents increased SBIRT knowledge, F(1, 167) = 32.1, p < .001. • No differences found between residency programs
Outcome goal 1 cont.Enhance residents’ knowledge of evidence-based SBIRT practices. Residency-specific SBIRT knowledge • Pediatrics increased SBIRT knowledge,F(1, 59) = 4.53, p = .038. • FCM maintained SBIRT knowledge, F(1, 46) = 2.2, p = .149. • IM maintained SBIRT knowledge, F(1, 61) = .20, p = .659.
Outcome goal 2Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients. • Readiness to use SBIRT (D’Onofrio et al., 2002) • Subscale of AES comprised of 7 10-point Likert scale items • Range: 0-100 • Sample item: “How ready are you to change your practice behavior to ask patients about quantity and frequency of their alcohol use?” • Confidence to use SBIRT (D’Onofrio et al., 2002) • Subscale of AES comprised of 7 5-point Likert scale items • Range: 0-100 • Sample item: “I am confident in my ability to discuss/advise patients to change their drinking behavior.”
Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients. • Readiness to use SBIRT: • No significant change in readiness from pre-training to 12 months post-training, F(1, 161) = .87, p = .353. • FCM reported higher readiness than IM overall, F(2, 161) = 4.7, p = ..010. • Pediatrics was not significantly different than the other two programs
Outcome goal 2 cont.Enhance residents’ readiness & perceived confidence to implement SBIRT with their patients. • Confidence to use SBIRT: • Residents overall reported higher confidence at 12-month, F(1, 161) = 27.3, p < .001. • FCM reported higher confidence overall than IM and Pediatrics, F(2, 161) = 8.1, p < .001. • Pediatrics was not significantly different than the other programs
Outcome goal 3Increase residents’ implementation of SBIRT practices with their patients. • Self-report of current SBIRT practice (D’Onofrioet al., 2002) • Subscale of AES comprised of 7 5-point Likert scale items • Range: 0-100 • Sample item: “How often do you formally screen patients for alcohol problems using brief screening tools (e.g., T-ACE, AUDIT, CAGE)?” • Pocket cards • Chart reviews
Outcome goal 3 cont.Increase residents’ implementation of SBIRT practices with their patients. • Current practice of SBIRT skills: • Residents overall reported higher current SBIRT practice at 12-month, F(1, 161) = 35.2, p < .001. • Significant interaction, F(2, 161) = 19.7, p < .001. • Both Pediatrics and FCM improved self-reported current practice . • Internal Medicine declined in self-reported current practice.
Summary of Resident Survey Data Findings • SBIRT core knowledge improved from pre-test to 12-month follow-up • Readiness to implement SBIRT did not change, but was high at pre-test • Confidence to use SBIRT improved from pre-test to 12-month follow-up • For self-report of SBIRT practice, residents overall improved from pre-test to follow-up • However, when departments were analyzed separately, Internal Medicine decreased from pre-test to 12-month
Outcome goal 3 cont.UTHSCSA Family Medicine Pocket Cards • Settings: • Family Medicine inpatient service at University Hospital in San Antonio, Texas • Subjects: • 285 adult patients, from July 2009 to May 2011. • Average Age: 47 • Gender Distribution: 71.3% Male
UTHSCSA Family Medicine Procedures • Patients were interviewed with a 4-step pocket card Step 1: Pre-screening questions for substance use Step 2: WHO ASSIST (Alcohol, Smoking, and Substance Involvement Screening Test) Step 3: ASSIST score to assess the level of risk and determine need for intervention Step 4: checklist describing the intervention, patient response, and future plan. • Residents were asked to complete 12 per year • 26 out of 26 trained residents participated • Residents completed 11 total on average
Step 2-3: ASSIST Results • 95.8% of patients screened positive for at least 1 substance • Avg. ASSIST Score was 19 indicating a moderate risk of substance abuse
Brief Interventions • When the ASSIST Score recommended a brief intervention, residents reported some form of brief intervention 69.4%(over two thirds ) of the time • Residents most likely to discuss consequences of use if ASSIST Score recommended brief intervention (79% of the time) • 8% of patients declined to discuss their response to screening
Referrals to Treatment • When the ASSIST Score suggests a referral to treatment, residents referred a patient to treatment 71.8% of the time • Residents were most likely to contact an LCDC (Licensed chemical dependency counselor) when ASSIST Score recommended a referral to treatment (46.5% of the time) Referrals to Tx