Using GAIN Data for Program Evaluation and Program Planning

1. Using GAIN Data for Program Evaluation and Program Planning Barbara Estrada, GAIN Senior Clinical and Evaluation Consultant, Chestnut Health Systems Kristin Hedges, Research Specialist Senior, University or Arizona, SIROW

2. Overview Resources for using the GAIN for Program Evaluation An introduction to a variety of tools available to help evaluators Resources for using the GAIN for Program Planning An introduction to how GAIN data can be useful for program planning Using GAIN data in the field A real world example of how one program has used GAIN data to help their program.

3. What will you do with your GAIN data?

4. Resources and Tools ftp Common Site Electronic Encyclopedias GAIN Evaluation Manual Evaluator Or Analyst Analytic Training Memos Tools for Collaboration Reports, Publications and Presentations Site Profiles SPSS Syntax

5. Program Evaluation

6. FTP Common Site * See the Evaluator Resource Sheet ftp://data.chestnut.org/ Username: Common Password: public

7. General Individual Severity Scale (GISS) Substance Problems Scale (SPS) Internal Mental Distress Scale (IMDS) Somatic Symptoms Index (SSI) Substance Issues Index (SII) Depression Symptom Scale (DSS) Substance Abuse Index (SAI) Homicidal/Suicidal Thoughts Scale (HSTS) Substance Dependence Scale (SDS) Anxiety/Fear Symptom Scale (AFSS) Traumatic Stress Scale (TSS) Behavior Complexity Scale (BCS) Crime and Violence Scale (CVS) General Conflict Tactic Scale (GCTS) Inattentiveness Disorder Scale (IDS) Property Crime Scale (PCS) Hyperactivity-Impulsivity Scale (HIS) Interpersonal Crime Scale (ICS) Conduct Disorder Scale (CDS) Drug Crime Scale (DCS) Scales and Indices: Psychopathology Case Mix * The GAIN-I has over 103 scales and indices

8. Scale and Index: Interpretative Cut-points Many scales and indices have cut points which define low, moderate and high clinical significance (severity). The names are the same as the scale or index name with a “g” for group at the end. Useful for defining need at both the client and program level. Based on: DSM or other clinical standards where available (e.g., clinical is 3+; 7 dependence) 50th & 90th percentile for common issues (e.g., days of alcohol use) 1+ and median of 1+ for zero saturated (more than half) and right skewed variables Reverse-coded if “up” is low clinical significance (e.g., Treatment Motivation)

9. GAIN Scales and Variables File This Excel spreadsheet contains a listing of the major scales and indices from the GAIN-I scale name, variable name, time period, GAIN-I items used, number of items, pages, scale type and cut points, purpose, short description, interpretation, references, syntax, full text items, and additional comments Scales and Variables File There is one for the GAIN-Q too Located at ftp://data.chestnut.org in: Evaluator Folder/Data Summaries and Reports/

10. GAIN Evaluator Manual • A learning tool for using GAIN data, GEM provides information about our usual procedures and examples from recent presentations or publications, as well as detailed information with specific syntax. This represents an option for analysis that is pragmatic, robust and cost-effective, but is not intended to be exhaustive or to represent the only option. • Evaluators can use the GEM to understand their GAIN data, plan analyses, get answers to common questions, see examples of syntax, and access key documents that make up the numerous appendices. • Located at ftp://data.chestnut.org Evaluator Folder\GAIN EvalManual

11. GAIN Crosswalk A version-specific listing of each GAIN item or scale with information about its status as a required or optional item and whether optional items are treatment related or recommended; and information regarding the purpose for/source of the item and any comments regarding its use. Evaluators can use this crosswalk to determine item comparability across study (optional items have lower available Ns) and reasons for specific items. Helps in selecting which version of the GAIN to use. GAIN Crosswalk 5.5 Located at ftp://data.chestnut.org in: Evaluator Folder/Data Summaries and Reports/

12. Multisite Collaboration Guidelines • An overview of the annual datasets prepared by Chestnut Health Systems including their purpose, inclusion rules and expectations. • Evaluators should use the guidelines and expectations included in this document to help their site establish protocols that will result in their meeting all inclusion rules and strive toward meeting all expectations. • Located at ftp://data.chestnut.org Data Management Folder/Data Manager Required Reading

13. Data Request Topic Summary • A list of other evaluators and researchers who have requested to use GAIN data, including their name, contact information, abstract title, and key topics. • Evaluators can use this list to find others who are examining a particular topic of interest and develop a collaboration or discussion related to earlier work and findings. • Located at ftp://data.chestnut.org Evaluator Folder\Requesting and Using GAIN Data

14. Annual Summary Analytic Slides • Four sets of Power Point slides using summary analytic data for all records, adolescent sites, justice sites, or adolescents (12-17). Includes characteristics, placement, treatment, and outcomes (NOMS) by gender, age group, race, treatment type and program. • Evaluators can use these slides (with acknowledgement) for reports, presentations or articles related to their analyses. • Conceptualize how GAIN data is used • Located at ftp://data.chestnut.org Evaluator Folder\Data Summaries and Reports

15. Adult and Adolescent Norms and Psychometrics Provides intake N, Mean, sd, alpha, percent by severity group (Lo/Mod/High), follow-up alpha and follow-up ICC for 3 groups of clients (adult, young adult, adolescent). In addition, it includes intake N, Mean, sd, alpha for adolescents by gender, age group and race. Evaluators can use the norms and psychometrics in the study planning phase or to describe or compare populations in reports to stakeholders, manuscripts, or other documentation. Psychometrics Located at ftp://data.chestnut.org Evaluator Folder\Data Summaries and Reports

16. Analytic Training Memos Series of documents created to help researchers overcome specific problems associated with the collection and analysis of data generated during substance abuse treatment research. Evaluators can use these memos to conduct analyses in the same manner as the GCC and/or to report on data handling for an article or presentation They are available free for download Located at ftp://data.chestnut.org Evaluator Folder\Training\Analytic Training Series

17. Syntax and Template Files SPSS syntax to read ABS exported data into SPSS, label variables and values, calculate scales, indices, grouped severity variables and other variables, and convert cases to variables. Evaluators can use these files to recreate data files any time in the same manner as the GCC. Located at ftp://data.chestnut.org Evaluator Folder\Syntax Files

18. Program Planning

19. Follow-up and Treatment Log Reports Reports generated in ABS that are used to help track site information, participants who are administered GAIN assessments, and the treatment they've received Follow-up rates, tracking treatment, etc. Use beyond grant requirements Provide feedback loop to treatment program staff and researchers Understand referral sources, discharges, transitions in treatment

20. Using Site Profiles Profiles include: Demographics; substance use data; Comorbidity data; risk data; treatment information; selected outcomes; individual site graphs, and two-site comparison graphs Useful to understand how your site compares to others Site Comparisons Profiles are updated quarterly for all programs, posted on APSS/GPSS site and e-mailed to each PI Located on the APSS website http:/www.chestnut.org/li/APSS/ OR http:/www.chestnut.org/li/APSS/under your specific program

23. What does the data tell us? • Understand the characteristics of your clients. • Are you aware of the cultural norms of the major ethnic subgroups of clients? • Do your staffing patterns reflect the client’s characteristics? Language, cultural background, etc.? • Does this data suggest investigating possible language barriers clients may be experiencing? • Look Deeper: • Items A3b1-5 (English proficiency) and XADMb (How GAIN was administered). • Item B9a5: help needed with language, religious, cultural or ethnic issues

25. What does the data tell us? • Lower (or decreasing) percentages in readiness to change, perception of the need for treatment and perception that alcohol and other drugs is a problem may necessitate the inclusion of a more, stronger, or targeted MET/MI interventions. • Lower problem orientation may be related to learned helplessness, severe depression, or weak decision making skills that may necessitate the inclusion of life-skill programs. • Look Deeper: • What are the characteristics of these clients?

27. What does the data tell us? • Work and school are also part of client’s recovery environment. • Higher (or increasing) percentages in work, school, criminal justice involvement will pose barriers to treatment. • Consider packaging or adding a concrete group or class (“Getting off of Probation/Parole”) that clients will see as relevant and useful in helping to reach goals while still getting recovery messages to them. • Does the program have strong ties to criminal justice agencies and do both agencies work collaboratively to reduce barriers to treatment (attendance in school/work versus attendance in treatment/meeting CJ requirements)?

28. Other ways to get help • Use our e-mail support lines for questions with: • general issues: gaininfo@chestnut.org • evaluation/analysis: gaineval@chestnut.org • data submission/data: datasubmit@chestnut.org • administration/certification: gainsupport@chestnut.org • software/web application: abssupport@chestnut.org • This presentation is available on your flash drive

29. Using GAIN data in the field

30. Data Snippets • Regular inclusion of data • Monthly reports • Monthly meetings • Clinical meetings • Benefits • Easier digestion • Importance of data • Improving ‘real time’ treatment

31. Sample Monthly Report

32. Sample Monthly Report Cont.

33. Data reports • Regular review of what data is saying • Quarterly / Biannually • Distributed to all project members • Reviewed in person to do a ‘walk through’ • Benefits • More in-depth look at site data • Brainstorm session over explanations • Improving ‘real time’ treatment

34. Interim Report-Year one • Report Sections • Introduction • Baseline Sample • Demographics, substance use, institutional involvement, mental health, home environment, social environment • Outcome Sample • Substance use, Mental Health, Home Environment, Social Environment • Summary Statements

35. Interim Report- Baseline Sample

36. Interim Report- Baseline Sample

37. Interim Report - Baseline Sample

38. Interim Report - Baseline Sample

39. Interim Report - Baseline Sample

40. Interim Report - Outcome Substance Use

41. Interim Report - Outcome Mental Health

42. Interim Report - Outcome Sample

43. Interim Report - Outcome Sample

44. Project Specific Site Profileby Dr. Josephine Korchmaros • Tailoring data using site profile template • Designed for our TCE/HIV project • Specific to project and goals • Easy to use format

45. TCE/HIV Site Profile

46. TCE/HIV Site Profile