State Agency Reform: The Experience of the Massachusetts Department of Youth Services

1. State Agency Reform: The Experience of the Massachusetts Department of Youth Services

2. Introductory RemarksTarek AnandanDirector, NDTAC

3. Evaluating N or D Programs • The What • Formative evaluation • Summative evaluation • The Why • Legal requirement • Program quality improvement • Program expansion/contraction • The How • Execution - internal or external • Funding – Federal, State, local, foundations

4. Evaluating N or D Programs “Nationwide, there is a scarcity of literature specific to effective or ‘best practice’ juvenile justice education, or, for that matter, systemic reform of similar systems” Evaluation of the Department of Youth Services Education InitiativeUMass Donahue Institute, 2008

5. Evaluation and Title I, Part D SEC. 1431. Program Evaluations (a) SCOPE OF EVALUATION—Each State agency or local educational agency that conducts a program under subpart 1 or 2 shall evaluate the program, disaggregating data on participation by gender, race, ethnicity, and age, not less than once every 3 years, to determine the program's impact… (d) EVALUATION RESULTS—Each State agency and local educational agency shall— (1) submit evaluation results to the State educational agency and the Secretary; and (2) use the results of evaluations under this section to plan and improve subsequent programs for participating children and youth

6. Evaluation and Title I, Part D Funding • Evaluation is an allowable use of Part D funds • State agencies and local education agencies can use a portion of their granted funds to support evaluation activities • Funds used for evaluation purposes should be reasonable in respect to the amount of funding that goes to the provision of educational services • State education agencies are allowed to reserve the greater of 1% or $400,000 for administrative purposes, including evaluation activities of SEA administrators

7. Evaluation Resources • U.S. Department of EducationUnderstanding Evaluation: The Way to Better Prevention Programshttp://www.ed.gov/PDFDocs/handbook.pdf • Bureau of Justice AssistanceCenter for Program Evaluationhttp://www.ojp.usdoj.gov/BJA/evaluation/ • Justice Research and Statistics AssociationProgram Evaluation Briefing Serieshttp://www.jrsa.org/pubs/juv-justice/index.html • “Approaches to Assessing Juvenile Justice Program Performance” • “Hiring and Working With an Evaluator” • “Strategies for Evaluating Small Juvenile Justice Programs”

8. Presenters • Christine KenneyDirector of Educational ServicesMassachusetts Department of Youth Services • Lonnie KaufmanResearch ManagerUMass Donahue Institute • Steven EllisResearch ManagerUMass Donahue Institute

9. State Agency Reform: The Experience of the Massachusetts Department of Youth Services Jane Tewksbury, Commissioner Edward Dolan, Deputy Commissioner Today’s Presenters Christine Kenney, Director of Educational Services, DYS Lonnie Kaufman, Research Manager, UMass Donahue Institute Steve Ellis, Research Manager, UMass Donahue Institute

10. DYS Vision The Department of Youth Services is a leader in the field of juvenile justice that collaborates with youth, families, communities, government, provider agencies, and others to prevent juvenile crime. We provide the most innovative and effective treatment and skill development services available to the youth committed to our care. The Department exemplifies diversity in the management of all our work and relationships. We effectively manage the resources necessary to achieve the Department’s mission.

11. DYS Functions DYS is the Commonwealth’s Juvenile Correctional Agency • Functions • Detention • Residential Treatment • Parole/Community • Each Year: • 18,000 to 20,000 arraignments • 5,700 bail admissions with average length of stay of 17 days • 1,200 commitments with average length of stay of 2.5 years • On any given day there are 2,000 committed youth in custody. • 49% in Residential Placements • 51% in Community Supervision

12. Building Educational Capacity Educational improvement efforts were spurred by the findings of the MA DYS 2001 Legislative Report Key Findings Related to Educational Services included concerns regarding: • Teacher Recruitment and Retention • Transitions • Special Education • Learning Methods, Curriculum and Other Resources • Vocational Education and Job Training • Computers and Technology • Infrastructure

13. DYS Reform Strategies 2001-Present • DYS in close collaboration with the Commonwealth Corporation and the Hampshire Education Collaborative, initiated a multiple phase education reform effort for DYS educational programs • Develop capacity to support education reform across the agency • Align all education programming to MA Curriculum Frameworks • Develop an effective community placement and transition process • This initiative reflected a collective commitment to meet state and federal standards for educational programming by developing a better coordinated, appropriately resourced educational system • In spring 2006, DYS initiated a system-wide evaluation to help measure progress and to guide further education improvement

14. Evaluation of DYS Education InitiativeWhy and How? • DYS wanted to revisit the original legislative report and assess change in areas that had been targeted for improvement • Minimum education requirements had provided a preliminary, but limited, view of the Education Initiative’s progress • A comprehensive evaluation would encompass a broader and more systemic view of the Education Initiative • Talked to local universities about how to approach • DYS made a commitment to allocate funding for evaluation for FY07/08 and issued an RFP • RFP was based on a logic model which guided core education strategies and activities • University of Massachusetts Donahue Institute was awarded the evaluation contract

15. Evaluation: Phase One Phase One: June 1 through September 30, 2006 • Literature Review • Preliminary Fieldwork and Evaluation • Interviews with key leaders of DYS, CommCorp and HEC • Site visit interviews with teachers and coordinators at eight programs • Attendance and participation at various system events • Review and cataloging of key system documents • Development of a Comprehensive Evaluation Plan • Presentation and Discussion of Preliminary Evaluation Findings

16. Evaluation Plan Evaluation Guided by Three Core Research Questions • To what extent has DYS successfully implemented strategies designed to improve the educational and life outcomes for youth they serve? What factors have influenced the extent or success of implementation? • Have the Education Initiative's core strategies led to, or are they leading to anticipated short or mid term impacts? What other impacts or outcomes have resulted from the Education Initiative? • Based on key findings of the evaluation, what should DYS focus on over the coming years in order to realize the long term goals/outcomes of the Education Initiative?

17. Evaluation Plan The Evaluation Also Identified: • Emerging promising practices • Causes and potential solutions for mixed finding and impacts • Causes and potential solutions for gaps in evidence, including additional data which need to be collected to properly measure certain outcomes outside the realm of the evaluation • Apparent gaps in the current Education Initiative • Key lessons learned through the implementation of the Initiative

18. Evaluation: Phase Two Phase Two: October 1, 2006 through August 31, 2007 • Several major data collection and analysis activities: • A DYS teacher survey that gathered the perspectives of over 90% of respondents • In-depth interviews of approximately 50 educators working at 12 DYS Education programs • Collection and analysis of all available institutional data • Case studies of six “featured practices” resulting from the Initiative • A second, targeted review of literature • On-site observations of selected programs and events, including statewide professional development

19. Top-Level Evaluation Findings • Findings show remarkable progress in the development and implementation of key interventions, and positive initial impacts. Evidence shows important systemic changes to: • Educational service delivery • Support of student transitions • Overall system management • Important new programs, policies, and infrastructure are now in place, which have facilitated improvements in the stability and quality of DYS’s: • Teacher workforce • Instructional resources and methods used in DYS classrooms • Operative culture in which teachers and students function each day • Continued progress is likely, assuming DYS can sustain its vigorous investment of time, resources, and attention • Findings underscore not only the need to continue and expand improvement efforts in the coming years, but also the challenges associated with delivering educational services to adjudicated youth

20. Sample of Approach to Presenting Findings Topic: Educational Service Delivery Key Finding: DYS has dramatically improved the competitiveness of education staff salaries in order to attract and retain well-qualified teachers • Prior to the Education Initiative • A 2001 DYS Legislative Report identified “concern about the extraordinarily high turnover rate amongst DYS teachers and its effect on the quality of educational services. It concluded that high turnover and problems with recruiting qualified staff were primarily attributable to salary inequities between DYS and public school districts • In response • DYS implemented a more competitive salary scale that increased the minimum salary for bachelor’s-level teachers, bringing starting salaries in line with state averages. The new salary structure also offers compelling incentives for teachers to pursue full professional licensure • Looking ahead • As DYS continues to consider important reforms to its salary structure, it may need to offer higher salaries in key sub-regions where public school salaries are highest. Further, DYS may need to equalize teacher benefits and working conditions across program vendors

21. Implementation Findings • Educational Service Delivery • DYS has dramatically improved the competitiveness of education staff salaries in order to attract and retain well-qualified teachers • DYS has made substantial progress in the development and implementation of a system-wide curriculum approach supported with instructional guides, templates, and material resources. • DYS and its partners have implemented an extensive professional development system and instructional coaching model. • Transition Services • DYS has identified a need for more robust employment pathways for appropriate DYS youth and is actively piloting and assessing new programs, and taking other steps, to support this need. • DYS has improved the infrastructure to support student transitions within DYS and into academic settings in the community. New models are being pursued.

22. Implementation Findings • Education System Management • DYS has expanded its management capacity and educational expertise through use of external vendors – most notably CommCorp and HEC. • DYS has improved its education system coordination and has taken important steps toward establishing more effective system-level accountability.

23. Short-Term Impacts of the Education Initiative Outcomes remain preliminary; however, it appears to have resulted in several positive short-term impacts within DYS Education Perceptions of Education Quality • DYS teachers’ perceptions of the quality of education provided to DYS youth have markedly improved, and future expectations are for continued improvements.

24. Short-Term Impacts of the Education Initiative Workforce Stability • Data show positive impacts on the stability of the DYS teacher workforce over the past three years, including a decrease in teacher turnover and in August vacancy rates • Data also showed a substantial improvement in teacher qualifications

25. Additional Short-Term Impacts Changes in Instructional Practices • According to teachers, new curriculum tools and templates have made a positive impact on instructional practice, and instructional coaches’ reports indicate that teachers are now implementing recommended practices with increasing frequency and quality. • Teachers report that statewide PD has had a moderate to high impact on their instructional practice and student learning in the past three years. Data also suggest that instructional coaching helps many teachers reflect on their teaching style and encourages them to adopt new teaching strategies. Impact on Educational Culture • Data suggest a positive cultural shift has resulted from the adoption of education reform within DYS, and that a stronger professional culture has emerged within DYS Education. These trends are strongest among treatment programs, but evident system-wide.

26. Other Features of the Final Evaluation Report Strategic Considerations • A set of strategic considerations intended to provoke discussion and inform debate among DYS and its partners as they move into the next phase of education reform Featured Practice Case Studies • A section featuring six case studies that focus on new programs, tools, and processes developed through the Education Initiative • A statewide professional development system • A system-wide instructional coaching model • A process for effective piloting of youth employability programs • A work-based training program in partnership with Putnam Vocational Technical High School • A universal student transcript to support student credit recovery • A “Community Transition School” to support student re-entry into public schools Other Reports • Additional evaluation reports available include two literature review reports, and results from the DYS teacher survey administered in January 2007

27. Lessons Learned • Money, time, experience • More money would have allowed development of additional data and more in-depth research • Barriers • Data still inconsistent across the system; not always available in a useful format • Data often incomplete or not up to date • What DYS would do in the future • Look at what baseline data would be needed for future evaluations to ensuremore quantitative results • Look at specific initiatives that would be evaluated and tied to evidence based research practices

28. For Further Information Christine Kenney, Director of Educational Services Massachusetts Department of Youth Services Christine.Kenney@state.ma.us 617-960-3324 Lonnie Kaufman, Project Manager University of Massachusetts Donahue Institute lkaufman@donahue.umassp.edu 413-587-2419 http://www.donahue.umassp.edu