1 / 19

Lessons Learned from a Synergistic Program Evaluation Model Partnership

This presentation discusses the benefits, challenges, and lessons learned from a successful program evaluation partnership between a state agency and a university program. It highlights the development of Project Excellence, a program aimed at enhancing program evaluation, outcome measurement, and reporting processes.

edwardp
Download Presentation

Lessons Learned from a Synergistic Program Evaluation Model Partnership

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lessons Learned from a Synergistic Program Evaluation Model Partnership Between a State Agency and University Program Michael J. Leahy Michigan State University Program Evaluation Summit Salt Lake City, Utah September 25, 2008

  2. Agenda • Brief overview of MSU and MRS. • Typical collaborations between university programs and State Agencies. • Problems, issues and needs that led to the development of Project Excellence. • Benefits of funded partnership to both parties. • Project Excellence - scope of work and how it works. • Lessons learned and success factors for PE in the public rehabilitation program.

  3. Project Excellence:A Program Evaluation Partnership Michigan State University Project Excellence Michigan Department of Labor & Economic Growth-Rehabilitation Services

  4. Michigan State University Office of Rehabilitation and Disability Studies Program began in 1956 Masters degree program in Rehabilitation Counseling CORE Accredited Doctoral degree program in Rehabilitation Counselor Education REACH Program for professional development Ranked the #1 program in the country (tied with University of Wisconsin-Madison) by US News and World Report Rankings of graduate programs in rehabilitation counseling Long-standing, mutually beneficial relationship with the public rehabilitation program in Michigan

  5. Department of Labor and Economic Growth - Rehabilitation Services (MRS) • 1.3 million people with disabilities in the Michigan - 27,000 citizens with disabilities served at any given time by MRS counselors. • Over 7,600 customers placed in jobs in 2007. • Core principles - customer-centered, equity, advocacy, collaboration, accountability, and continuous improvement • 275 Rehabilitation Counselors. • Heavy urban and isolated rural areas. • History of innovation in service delivery • Highest unemployment rate in the nation at 8.9% • High turnover -- early retirements of leadership

  6. My Initial Background in PE and Programmatic Research • Crossroads Rehabilitation Center in Indianapolis Indiana • CARF Section 9 Standards on Program Evaluation (mid 1970’s) • Trained in Program Evaluation through national CARF initiative • Walker Model (Dr. Robert Walker) • Became a trainer for Region V RRCEPS in program evaluation (late 1970’s) • Worked in this capacity for a number of years with community-based organizations throughout the region

  7. Typical Collaborations between a State Agency and a University Program • Membership on Advisory Councils • Input into graduate training programs • Serving as adjunct instructors and guest lecturers • Practicum and Internship sites for students • Professional development for agency staff • One substantive area that is not typically an integral aspect of the partnership • is research and program evaluation • Given the level of accountability for outcomes, the complexity of the process, • and the limited resources of most state agencies to effectively devote time, • energy and and expertise to program evaluation, this appears to be an ideal • place for a university program to further contribute to the partnership.

  8. Problems Viewed as Opportunities MRS was experiencing a lack of stability in the program evaluation area - frequent turnover and difficulty finding and retaining qualified personnel were persistent problems. Agency very active in implementing a series of new strategic directions and programs and were unable to adequately measure the impact of these initiatives and use data for continuous improvement in performance. To have quality programmatic research with timely information and consistently evaluate programs over multiple years became a greater need and objective of the agency. Key factor - - State Director and Executive staff value of PE.

  9. Benefits of the Funded Partnership • MSU • Opportunity to provide a rich, real world, field based environment • to enhance the training of doctoral students in research and program evaluation. • Provided a unique revenue stream to fund doctoral students as Research • Assistants. • MRS • Stabilize and greatly expand its programmatic research and program evaluation • capacity to meet routine regulatory reporting requirements. • Use program evaluation for the first time, as a major component of its continuous • improvement strategy for the agency.

  10. Project Excellence:A Program Evaluation Partnership • Began in 2001as a three-year project designed to enhance MRS program evaluation, outcome measurement, and reporting processes. After seven years in operation, will enter into a new grant in October for another five years (2008-2013). • Administered by MSU Rehabilitation Program faculty, with two full-time doctoral level research associates, and six research assistants (doctoral students). • Collaborative and consultative approach to working with MRS administrators; counselors; technical, training, and support staff; and customers.

  11. Project Excellence:A Program Evaluation Partnership Project Goal Project Excellence will further develop and improve the application of qualitative and quantitative data analysis with respect to Michigan Rehabilitation Services culture and programs while prioritizing regulatory responsibilities and program excellence values.

  12. Regulatory Needs RSA 911 Database RSA Performance Indicators Customer Satisfaction Needs Assessment Director’s Requests MCTI Annual Performance Agency Priorities Business Services Michigan Works! Youth Services Soc. Sec. Recipients Minority Populations Special Requests from MRS Administrators and Senior Managers MRS Project Excellence MSU Project Identified Needs MSU Student and Faculty Research Protocols Project Excellence Scope of Work

  13. How the Individual Projects are Selected Each Year Each year the State Agency calls for ideas and concerns that PE could address from all levels of the organization. These ideas are then reviewed with the Execute staff to identify priority areas. Once these potential projects are prioritized the Agency and the PE staff meet to review and make decisions about the scope of work for the upcoming year. Each year, over the past seven years, PE has had between 10-14 individual projects specified within the scope of work.

  14. Examples of Projects • Annual review the RSA 911 Data • Data Integrity Study • Customer Satisfaction • Services to Minority Populations • Business Services • Re-organization Impact Study • MI Transition in the 21st Century • Effectiveness of MRS sponsored CRO Grants • Michigan Career and Technical Institute • Orientation Process Evaluation • Vendors Qualifications and Standards • MI Comprehensive Needs Assessment • Best Practices related to Successful Outcome

  15. Ongoing Projects Project Excellence Flow of Operations Implementation Team Senior Manager Input District Manager Input Implementation Team Review Training Dissemination Evaluation Special Projects Work Groups SMP Transition Youth Adjudicated Youth MI HI Connections Customer Satisfaction Business Services MCTI Grants

  16. Lessons Learned - Success Factors • Trust • Commitment at the top of the organization a must • Long history of collaborative efforts • Learning organization with common values • Creative leadership and management styles to address implementation of findings from PE - continuous improvement. • Supportive Michigan Rehabilitation Council • Enjoyment of projects, collaboration, and partnership • Common goal to improve service delivery and enhance employment outcomes • Common commitment to provide a variety of research opportunities to doctoral students

  17. Doctoral StudentExperiences • What was gained • Positive influences on development as researchers • and leaders • Approaches to team work on research—from concept • to product • Methods for increasing research skills via access to • relevant data—beyond 911 data.

  18. Final Thoughts • Integration of PE into the fabric of the organization. • Inclusive style of participation on design and implementation teams. People need to feel involved -- not an audit function, but part of a continuous improvement strategy for the agency. • Clear communication of findings and implications for all levels of the organization. • Go beyond the analysis of passive data to design and carry out experimental and quasi-experimental research on specific interventions -- evidence-based practices.

  19. Final Thoughts • Seek out other program Evaluation Specialists in State Agencies and other human service institutions to share ideas, problems and challenges. • The PE Summit here in Salt Lake City is a wonderful start to develop a learning community among professionals with similar roles and responsibilities throughout the nation. • It is my hope that this group will continue meeting and communicating in the future. • Best of luck to all of you!!

More Related