1 / 39

Measuring Changes in Reclaiming Futures Communities: National Evaluation Results

This report presents the national evaluation results of the Reclaiming Futures initiative, focusing on measuring changes in substance abuse interventions in juvenile justice. The evaluation tracked change in 10 different service systems with 10 different change strategies using bi-annual surveys. The report includes survey results for various indices, such as access to services, data sharing, systems integration, and treatment effectiveness. It also examines bias in survey responses and the influence of insider involvement in the initiative.

bbadillo
Download Presentation

Measuring Changes in Reclaiming Futures Communities: National Evaluation Results

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Changes in Reclaiming Futures Communities: National Evaluation Results Annual Meeting of the Coalition for Juvenile Justice Washington, DC June 11, 2007 Jeffrey A. Butts Chapin Hall Center for Children University of Chicago Views expressed are those of the author and should not be attributed to the University of Chicago, Chapin Hall Center for Children, their trustees or funders..

  2. Reclaiming Futures • An initiative of Robert Wood Johnson Foundation • Focused on substance abuse interventions in juvenile justice • Principal strategy was organizational change/system reform • Evaluation challenge: Tracking change in 10 completely different service systems with 10 different change strategies

  3. Surveys of Reclaiming Futures Communities • National evaluation tracked change with bi-annual surveys • Surveys measured quality of systems as reported by 20 to 40 expert informants in each community • Six surveys conducted in: 2003 – December 2004 – June and December 2005 – June and December 2006 – June • Response rate averaged 70 percent and ranged from 63% (December 2003) to 73% (December 2004)

  4. Surveys of Reclaiming Futures Communities • 58 questions about quality and effectiveness • All questions were brief statements • Respondents indicated whether they strongly disagreed, disagreed, were neutral, agreed, or strongly agreed • Answers coded so higher scores indicate positive opinions • Responses scored: -10 strongly negative -5 somewhat negative 0 neutral +5 somewhat positive +10 strongly positive

  5. Surveys of Reclaiming Futures Communities

  6. Surveys of Reclaiming Futures Communities The overall effectiveness of treatment was measured with five statements: In the past three months, … 1. The substance abuse treatment needs of youth in my community were adequately met. 2. The mental health needs of youth in my community were adequately met. 3. Graduated sanctions were used effectively to support treatment goals for youth. 4. Youth-serving agencies in my community generally did a good job serving youth. 5. Youth-serving agencies in my community were usually able to provide youth with the range of services they needed.

  7. Surveys of Reclaiming Futures Communities

  8. Survey Results for Each Index

  9. ADMINISTRATION INDICES Access to Services

  10. ADMINISTRATION INDICES Data Sharing

  11. ADMINISTRATION INDICES Systems Integration

  12. ADMINISTRATION INDICES Resource Management

  13. COLLABORATION INDICES Client Information

  14. COLLABORATION INDICES Partner Involvement

  15. COLLABORATION INDICES Agency Collaboration

  16. QUALITY INDICES AOD Assessments

  17. QUALITY INDICES Treatment Effectiveness

  18. QUALITY INDICES Targeted Treatment

  19. QUALITY INDICES Cultural Integration

  20. QUALITY INDICES Family Involvement

  21. QUALITY INDICES Pro-social Activities

  22. Change in scores across all Reclaiming Futures communities between December 2003 and June 2006, ranked by percentage of possible improvement realized Percent of Possible Improvement Realized Rank Survey Index Statistically Significant? Note: The total improvement possible in any index is limited by the score in Survey 1. Indices with high values in Survey 1 have less room to improve, given that +10 is the maximum possible score. “Possible improvement” realized was calculated by taking the total difference in survey scores (Survey 6 – Survey 1), and dividing by the maximum possible improvement, or 10 minus the first survey score. Change Indices are rank ordered from largest to smallest change, based on percentage of possible improvement realized. A t-test was used to determine each if change was significantly different from zero.

  23. Significant increases in survey indices by RF community, 2003-2006 Note: Significant increases occurred when the difference in mean index scores on the first survey (December 2003) and last survey (June 2006) was statistically significant ( p < .05). An increase was considered to be linear when at least 4 of 5 possible score changes between the six surveys were in a positive direction.

  24. None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

  25. What About Bias? • Answers to survey questions could be measures of self-interest or wishful thinking • Regression analyses were performed to check on this possibility • Analyses tested the influence of several factors on the trajectory of changes in individual survey scores • Main variable of interest: Did survey scores change more for RF “Insiders” -- the 4 or 5 people in each community who were involved in planning the RF initiative and who were closer to RWJF?

  26. Most Indices Were Not Affected by Bias

  27. Treatment Effectiveness The values differ, but the trajectories are positive for both groups, indicating growing confidence in the effectiveness of treatment.

  28. Treatment Effectiveness

  29. AOD Assessment

  30. A Few Indices Were Affected by Bias

  31. Cultural Integration

  32. Partner Involvement

  33. If we cross out the indices that were possibly affected by bias… None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

  34. None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

  35. None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

  36. None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

  37. None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

  38. Conclusions • Respondent ratings of local juvenile justice systems improved significantly between 2003 and 2006. • Increases were statistically significant in 12 of the 13 survey indices. • Improvements were especially dramatic in the ratings for: - treatment effectiveness, - the use of client information in support of treatment, - the use of screening and assessment tools, and - overall systems integration. • These findings suggests that the coordination and quality of substance abuse interventions for youthful offenders improved during the RWJF Reclaiming Futures initiative.

  39. For more information: Jeffrey A. Butts Research Fellow Chapin Hall Center for Children University of Chicago 1313 East Sixtieth Street Chicago, IL 60637 (773) 256-5163 jabutts@uchicago.edu This presentation can be accessed at: www.jeffreybutts.net

More Related