Successes and sustainability: Lessons learned from the Oklahoma COSIG evaluation

1. 1 Successes and sustainability: Lessons learned from theOklahoma COSIG evaluation 5th Annual COSIG Grantee Meeting, May 28-30, 2008

2. 2 Presenters Andrew L. Cherry, DSW, ACSW Oklahoma Endowed Professor of Mental Health, University of Oklahoma, School of Social Work, Tulsa Campus, OU OK-COSIG Project Evaluator L. D. Barney, LADC, Oklahoma Department of Mental Health and Substance Abuse Services Co-Occurring Program Specialist This project was supported by funding awarded by the Oklahoma Department of Mental Health and Substance Abuse Services (ODMHSAS) by SAMHSA. Points of view in this document are those of the author and do not necessarily represent the official position or policies of ODMHSAS.

3. 3 Successes and sustainability Moving science to service, in a way that is sustainable, is more about changing the organization than it is about the science or the services provided. The Oklahoma local evaluation was designed to identify and document sustainable change. In this design, the Oklahoma local evaluation was conducted at three levels. Macro Mezzo Micro

4. 4 The Three Levels Evaluation Macro: This level focused on changes at the state department level. Changes in the organization and changes in the Rules. Mezzo: This level focused on agency changes. Micro: This level focused on the individual seeking treatment; the outcome of treatment for people with a co-occurring disorder versus standard treatment.

5. 5 Sustainability Requires Intrinsic Organizational Change Scientific discoveries can become a reason for changing services. Even so, the science cannot be the how of changing. No one in the behavioral health fields disagree that best practices should be provided to people seeking mental health and addiction treatment. The conundrum, however, is how to make the changes in elaborate systems; especially, large complex elaborate systems that are sustainable.

6. 6 Stages of Organizational Change in Complex Systems Stage 1: Unfreezing. An initiating condition is needed that creates stress related to current operating procedures. Stage 2: Disequilibrium. The initiating condition increased stress to the point that the level of stress exceeds the capacity of the System to absorb the stress; disequilibrium occurs. Stage 3: Reestablishing Equilibrium. Forces converge and the organization is transformed into a new configuration that can accomplish the additional work. Equilibrium is reestablished.

7. 7 Tools and skill sets for managing change in complex systems Stage 1: Unfreezing Objective 1: Raising Awareness Use government monographs and documents, professional publications, and experts to raise awareness. Documents such as TIPS 42 and other scientific materials are available. Technical assistance from the Federal level gives the project added credibility. Regional information meetings, conferences, workshops, and town hall meetings can be used as platforms to raise awareness. Charismatic consultants can help orient and motivate clinicians, administrators, and other shareholders. Explain the benefits that will result from the changes, and provide ideas and approaches to implement the new treatment modalities.

8. 8 Tools and skill sets for managing change in complex systems (cont 2) Stage 1: Unfreezing Objective 2: Develop support for the needed change Use consensus building activities and skills to engage and retain stakeholders in a dialogue about needed change. Collection of data on opinions, committing to shared responsibility and accountability, and open planning Rather than hold up or slow down change opportunities so that goals can proceed linearly as planned, move goals and their components forward that met with the least resistance such as: consultant activities, shareholder committee activities, staff training activities, and changes in legislative rules. Accomplishing these two objectives in Stage I meant convincing some agency shareholders that services and treatment for people with a co-occurring disorder needed to be different than the standard treatment for people with a mental illness or a substance use disorder.

9. 9 Tools and skill sets for managing change in complex systems (cont 3) Stage 2: Disequilibrium Objective 1: Collaboration Network and capacity building strategies can be used to facilitate the exchange of ideas, skills, and resources among agency administrators and clinicians. An implementation plan that is developed by shareholders or that the shareholders agree on is crucial. The tools that are the easiest to use during this stage are related to technical assistance and training. The more difficult tools are associated with implementing the delivery of the new services.

10. 10 Tools and skill sets for managing change in complex systems (cont 4) Stage 2: Disequilibrium Objective 2: Motivation Continued workshops, town hall meetings, and trainings are useful tools for maintaining and fueling motivation . Make available skills training for clinicians on how to provide the new services; and training on ways to make the administrative changes needed to support the new services. Small incentives ($15,000 in the first year and $5,000 in the second year) were used to motivate participating agencies to meet specific goals: completing the fidelity instrument, sending staff to trainings, and sending staff members to the OK-COSIG Committee meetings. Maguire and McKelvey (1999) described this phase of change as adaptive tension.

11. 11 Tools and skill sets for managing change in complex systems (cont 5) Stage 2: Disequilibrium Objective 2: Motivation Even though viewed as a positive change by the vast majority of the shareholders, at the same time, change is an impediment and interferes with the need of agency administrators to develop yearly operational plans, budget and allocate resources, and meet contractual obligations to funders and clients. This is a formidable counterforce. From this pull and push period a compromise emerges. This compromise occurs among members who know they must make changes and have the freedom to change in ways that tend to meet the goals of the innovation and do the least damage (caused by the change) to their agency operations as a whole.

12. 12 Tools and skill sets for managing change in complex systems (cont 6) Stage 3: Reestablishing Equilibrium Objective 1: Maintenance The primary tasks are to sustain the change, set standards, and reorient the culture. Once the changes are taken as far as they can go, the focus must turn to activities that support the changes made, and establish the changes as a part of the organization/agency culture (Buchanan, et al., 2005). To sustain the changes that have been made and to promote the changes that will still be in progress, ongoing initiatives to expand the base of support for the changes are needed. Maintaining a presence and a voice for co-occurring programming at the State policy level is needed.

13. 13 Tools and skill sets for managing change in complex systems (cont 7) Stage 3: Reestablishing Equilibrium Objective 2: Monitor performance The monitoring of agency services and client outcomes uses fairly concrete tools. These tools need to be developed from existing measures and agreement on new measures. In this case example, there were three general areas to monitor. They were: policy compliance, program fidelity, and client outcomes based on the Individual Client Information System (ICIS) database.

14. 14 Tools and skill sets for managing change in complex systems (cont 8) Stage 3: Reestablishing Equilibrium Objective 3: Reorient the Organizational Culture A tool that is extremely useful was the empowerment of a committee called the �Regional Change Agent Committees.� The task of this group of clinicians is to continue to support and disseminate best practices in their region. State level recognition of successful changes that were accomplished is useful for moving the trajectory of agency focus. New employee orientation materials and training that includes clinical knowledge and best practice approaches for treating people with co-occurring disorders will help make and sustain cultural changes within those organizations.

15. 15 The Macro Level Evaluation At the end of year three (3), the implementation of a standard practice for screening and assessment for people with a co-occurring disorder became new language in the rules governing mental health treatment (Title 450:17, Chapter 17) and substance abuse treatment (Title 450:18, Chapter 18). These rules specify that treatment provided by mental health treatment providers and substance abuse treatment providers include treatment for people with a co-occurring disorder. The protocol covers: Welcoming Practice Guidelines, Screening Practice Guidelines, Assessment Practice Guideline, and Integrated Recovery Plan Practice Guidelines.

16. 16 The Mezzo Level Evaluation The OK-COSIG team achieved agreement on the needed change using consensus building activities and skills. Scientific findings and new best practices were introduced as a reason to change. A core level training was developed and delivered. The training was provided to all participating agency staff, both professionals and non-professionals. The focus of the training was on engagment, screening, assessing and using integrated treatment approaches with people who have co-occurring disorders.

17. 17 The Mezzo Level Evaluation The OK-COSIG team achieved agreement on the needed change using consensus building activities and skills. Scientific findings and new best practices were introduced as a reason to change. A core level training was developed and delivered. The training was provided to all participating agency staff, both professionals and non-professionals. The focus of the training was on engagement, screening, assessing and using integrated treatment approaches with people who have co-occurring disorders.

18. 18 The Mezzo Level Evaluation (cont 2) Preliminary Findings: Differences between Model programs and Control programs on identifying people with a co-occurring disorder (N = 19,241).

19. 19 The Mezzo Level Evaluation (cont 3) Preliminary Findings: Differences between Model programs and Control programs on Treatment Completion (N = 19,241).

20. 20 The Mezzo Level Evaluation (cont 4) Preliminary Findings: Differences between Mental Health Model programs and Control programs on Days in Treatment (N = 19,241).

21. 21 The Mezzo Level Evaluation (cont 5) Preliminary Findings: Differences between Substance Abuse TX Model programs and Control programs on Days in Treatment (N = 19,241).

22. 22 The Micro Level Evaluation The prevalence of people with the co-occurring disorders (mental illness and substance misuse) is essential for service systems such as ODMHSAS and its contracting agencies to be able to provide effective treatment and services (McGovern, Xie, Segal, Siembab & Drake, 2006). Additionally, Drake and others (2005) have suggested that �future research should refine existing interventions and clarify a typology of dual diagnosis patients.� The following is a typology based on people who were admitted for treatment with an indication of a co-occurring disorder in FY 2005�2006 to 15 model agencies and 5 control agencies in Oklahoma.

23. 23 The Micro Level Evaluation (cont 2) Putting a face on people with a co-occurring disorder One of the overarching impressions, based on this data analysis, is that people with a co-occurring disorder have more in common with each other than they have differences. They are clearly an identifiable group of people and different from people who seek treatment with a mental health disorder or with an addiction. Age Differences: As a group people in this sample were approximately 36.5 years of age. Men and women with an indication of a co-occurring disorder tended to be younger. Relationships: People with a co-occurring disorder were more likely to be involved or were involved in a relationship than people with no indication of a co-occurring disorder. However, they reported more family problems on the CAR Family subscale and the ASI Family subscale. People with an indication of a co-occurring disorder were more involved in relationships and had more problems in their relationships.

24. 24 The Micro Level Evaluation (cont 3) Putting a face on people with a co-occurring disorder. Education: There was no significant difference in education among males; however, women with an indication of a co-occurring disorder had slightly less education than women without an indication of a co-occurring disorder. Income: The average yearly reported income for all men ($11,636) admitted to treatment was slightly higher than all women ($10,648) admitted for treatment. The per capita income in Oklahoma in 2006 was $32,391. There was no statistical difference between men and women with an indication of a co-occurring disorder and men and women without an indication of a co-occurring disorder. Homelessness: Both men and women with an indication of a co-occurring disorder were likely to be homeless. Among homeless people in this study sample approximately 50% were identified as having a co-occurring disorder.

25. 25 The Micro Level Evaluation (cont 4) Putting a face on people with a co-occurring disorder. Admission Status: You can expect both men and women with a co-occurring disorder to be admitted as a result of a legal intervention. Arrests: Men and women with an indication of a co-occurring disorder will have had more arrests when entering treatment. Drug Use: Adults with an indication of a co-occurring disorder reported using drugs more frequently. They more often reported a history of IV drug use. However, there was no significant difference between people with an indication of a co-occurring disorder and people with no indication of co-occurring disorder in terms of age of first drug use. Serious Mental Illness: People with a co-occurring disorder were less likely to be identified as having a serious mental disorder (men = 25%, women = 15%).

26. 26 Conclusions In the case of the OK-COSIG project, broadly speaking (without regard to specific details or exceptions) the effort to raise awareness was enormously successful. In fact, resources could have been better distributed in Stage I. Too much attention was given to tasks related to raising awareness. There was already a great deal of awareness among clinicians. Some of those resources would have been better expended to enhance shareholder involvement by African Americans, Native Americans, Hispanics, consumers, advocates, and other groups. Additionally, more resources committed earlier and focused on providing the knowledge and developing training on how agencies could change would have reduced the fragmentation during Stage II.

27. 27 Conclusion (cont. 2) As in any change effort, the OK-COSIG team was unable to change many of the traditions held dear in the large complex organizations. Even so, the OK-COSIG implementation team did accomplished a great deal. In Oklahoma today, people with a co-occurring disorder have a much greater chance of being identified and receiving treatment that is responsive to his or her co-occurring disorder. As a clinician observed during one of the evaluation focus groups, �Without the OK-COSIG project, this level of service and treatment for people with a co-occurring disorder would not have been available for years to come.�

28. 28 Limitations A large part of the evaluation is based on qualitative data: documents, committee minutes, key informant interviews conducted quarterly, and twelve focus groups over the last three years and six months. The quantitative data came from the Individual Client Information System (ICIS) . This is a state database that collects treatment information from agencies who contract with ODMHSAS. The ICIS weakness is that different people with different levels and commitment to accuracy input the data. The ICIS strength is that the data used to analyze the programs includes over 20,000 cases on 115 variables.

29. 29 Direction for Future Research The next steps: Compare FY 2005-2006 to FY 2006-2007 data. Compare 15 model agencies from year two to 13 new model agencies that came online in year three. Compare the ICIS data with the GRPA data. Improve typology using GRPA data in terms of previous treatment episodes. Compare the control programs from year two with year three to see if the Rules changes alone improved services provided to people with an indication of a co-occurring disorder.

30. 30 Hopefully the Beginning of Sustainable Change