davidr:

1. davidr: C-8 Moving Toward Excellence: Integrating Performance and Outcome Measurements A Data-driven Organizational Quality and Performance System Jill Pfitzenmayer, Ph.D. Child & Family Services, Newport, RI David Robinson, Ed.D. Center for Evaluation and Research with Children and Adolescents, Massachusetts Society for the Prevention of Cruelty to Children, Boston, MA

2. Presentation Objectives 1.   Participants will learn how to identify program goals, objectives and specific program activities toward the accomplishment of program mission, vision and values. 2.   Participants will obtain information about establishing an organizational structure that supports critical analysis of program performance and quality improvement. 3.   Participants will identify specific outcome indicators, measurement tools and data collection methods relevant to their program plans.

3. Presentation Objectives 4.   Participants will learn how to link staff training, employee retention, program performance and outcome measurement efforts. 5.   Participants will be able to understand when internal outcome assessment can be enhanced by the introduction of an outside evaluator as consultant or primary investigator.

4. Agenda • Conceptual framework • Description of the identified needs and problems • Discussion of the methodology and approaches used to address the systems issues, • Review of specific agency initiatives and reporting structures developed to enhance organizational quality, and • Discussion by an outside evaluator familiar with the agency who was brought in for consultation

5. Performance Measurement – “Ongoing monitoring and reporting of program accomplishments, particularly progress towards preestablished goals.”* • type or level of program activities (process) • direct products and services delivered (outputs) • results of products and services (outcomes) • Program Evaluation – “Individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working.”* • internal or external experts and program managers • learn benefits of program or how to improve it Definitions * GAO, April 1998

6. Integrating Performance Measurement and Program Evaluation: A Conceptual Framework • Linked to institutional mission, goals, objectives, strategic plan, process and outcome evaluation • SMART - Simple, Minimum paper, Actively used, Revised, and Transferable to new programs • Best Practices – continuously updated incorporating new findings, technology and resources • Comprehensive - Quantity, Quality, Effort and Effect

7. Program Evaluation

8. Performance Measurement System 1. Gilbert Performance Engineering Approach • Develop a MODEL of process, department, program • Define INDICATORS of valued results and expectations • Take MEASURES to see how well process is working • Implement METHODS of improvement and evaluate results

9. 2. Friedman Four Quadrant Approach Performance Measurement System QUANTITY QUALITY INPUT OUTPUT

10. Effective Organizational Performance Measurement Systems • Commitment – Leadership team committed to measurement • Clarity – Degree of clear strategies and metrics • Metrics – Indicators measure valuable processes and expectations • Alignment – Performance measures aligned with key people processes and structures • Involvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system

11. Information about CFS • Multi-service organization over 136 years old. CEO been in office for 22 years; powerful Board of Directors • Primary services: • Nine child & adolescent residential sites • Two child care centers • Family counseling program • Home-based program • School-based programs • Transitional housing program

12. Implementation of Continuous Quality Improvement (CQI) Program • Minimal COA requirements: • Chart review • Customer satisfaction • Utilization review in select areas • Individual outcome data • Indication of systematic review of agency

13. Implementation Model: Challenges • Problem 1: disconnect between accrediting body and agency leadership perceptions of need • Problem 2: global discomfort with/ suspicion of “science” and “data” • Problem 3: Continuous Quality Improvement is supposed to involve all levels of staff, not be a top-down process

14. Steps Required to Develop an Outcome System Year 1: • 1. Assemble committee representing key constituents of the agency: • Agency leadership • Information systems • Human Resources • Business Office • Program staff • Parent • Consumer

15. Steps Required to Develop an Outcome System Year 1: • Develop a conceptual model to help staff review program activities and outcomes (e.g., logic models) • Map out organizational structure and evaluation activities (e.g., department teams) • Identify strategy that makes sense to staff to collect, analyze and use information

16. Steps Required to Develop an Outcome System Year 1: • Develop protocol and process for all levels of staff participation • Develop tools for reporting program activities at least quarterly and identify survey instruments • Train staff on recommended model and process

17. Identify a Solution-Focused Process PLAN Decide what is the problem Brainstorm solutions DO Enact solutions Collect data on how solutions are working ACT Revise action plan if needed Discontinue plan Continue to monitor CHECK Analyze data Discuss findings

18. CQI Activities Related to COA Standards C&FS develops long-term strategic plan (4 yrs) (G2.3) Identified stakeholders participate in all appropriate activities (G2.2) Each dept. develops annual plan of goals and objectives (G2.4) Internal & external audits & reviews (G2.10, G2.5) Chart Reviews (G2.6, G2.1) Customer satisfaction surveys (G2.8, G2.10) Outcomes Measures (G2.7, G2.10) Quarterly review of each dept.achievements by agency CQI Team (G2.9) Reassessment, as necessary (G2.11) Annual reports of year-end progress(G2.9)

19. Flow of CQI Activities • Annual plan • Quarterly Monitoring summary • Summaries of client/referral satisfaction forms • Summaries of incidents that occurred over the quarter • Summary of external review activities • Review department-specific and aggregated findings

20. Year 2: Challenges to CQI System • Many staff did not understand logic models • Program Directors didn’t understand basic concepts (e.g., difference between benchmark and baseline data) • Challenges existed around grasping conceptual material and managing task of putting information on computer!

21. Year 2: Challenges to CQI System • Some programs never tracked outcome data at all • State-funded programs required their own outcome tools that were either cumbersome or lacked validity • Staff lacked buy-in, experienced turnover or couldn’t make use of findings • Several programs had a hard time identifying individual outcome indicators

22. Year 2: Challenges to an Outcome System • Mechanically tracking data and outcomes became a challenge—no standardized data base in the agency • Changes implemented as a result of outcome data were often not documented • Staff were “busy” but couldn’t document real client change (i.e., conflating outputs and outcomes)

23. Year 2: Small Successes for an Outcome System • Some programs used the process to study and implement real change • Residential Network: system for getting paperwork accomplished • Family Counseling: saw change in patient population via use of SCL-90-R • School-based: outcomes supported pilot program around prevention education

24. Creating Organizational Change • Year 2: Agency/Board leadership in strategic planning meetings, which required a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis of programs • Lack of careful investigation of program outcomes forced leadership to rely more on fiscal and output information than on outcome data • Frustration Innovation

25. Continuous Quality Improvement Reactive Minimal Board involvement Driven by COA requirements Voluntary management participation Organizational Quality & Performance Proactive Board subcommittee Driven by strategic planning process Sr VP assigned to job Creating Organizational Change

26. Coordination of OQP Activities Agency Strategic Plan Organizational Quality & Performance Assess performance and outcomes to align agency goals and program activities Learning Institute Link staff needs with training and learning activities Employer of Choice Initiatives Maintain focus on performance to attract and retain highest quality staff

27. Moving Toward Excellence: Changes in Culture and Systems • Staff are now required to identify steps in developing programs reflecting excellence and best practices • Management are held accountable for process and program outcomes • Systematic data collection and analysis requires new MIS system • Agency-wide committees and initiatives brought into oversight process

28. Moving Toward Excellence: Changes in Culture and Systems • New OQP committee members include Corporate Communications and Development in order to ensure inter-departmental coordination • Methodology for tracking process and outcomes in place • OQP SVP meets provides technical assistance in program plan development

29. Moving Toward Excellence: Goals of OQP Process • Department-specific process and outcome data will be collected, analyzed and acted upon • Inter-departmental coordination will improve • Corporate Com, MIS & Development included in program planning • Programs work together on mutual or complimentary projects • Inter-program conflicts will be quickly identified and resolved

30. Moving Toward Excellence: Goals of OQP Process • Communication across departments and up, down chain of command will improve • Agency leadership will have improved and speedier methods of determining where there are problems and strengths in programs • Agency and program leaders have tools to continuously plan for the future

31. CFS Program Performance Plan

32. CFS Program Outcome Plan

33. CFS Quarterly Program Performance Report

34. CFS Quarterly Program Outcome Report

35. Moving Toward Excellence: How Do Program Staff Use OQP System? • Program Directors use process to bring their vision into program planning • Staff are oriented to OQP performance and outcome plans upon hire as a learning tool • Program leaders use plans to maintain an eye on program improvement activities

36. Moving Toward Excellence: Lessons Learned • Appreciate what you bring to the process from your own discipline and theoretical orientation and how they inform your outcome assessment work • Ask for help. Use volunteers, experts, etc. • Get organized and don’t panic • Don’t underestimate the value of communication • Maintain a stance of being inquisitive, not judgmental • Remind yourself and your staff that the process is developmental • Aim for direction, not perfection

37. Genuine Unsolicited Testimony from Staff: “I wanted to share with you my excitement of having completed the baseline ECERS and ITERS for child care today!…With this data we can now create our baseline and use it to establish our movement forward to improved overall quality in our Centers for the children, families and our staff! This is so exciting. Good things are in our future, along with much hard work…I am confident that I will lead the way to our success!” Patty Shelley, Child Care Program Director

38. ASSESSMENT: Case Study of CQI & Performance Measurement System • Commitment – Leadership team committed to measurement • Clarity – Degree of clear strategies and metrics • Metrics – Indicators measure valuable processes and expectations • Alignment – Performance measures aligned with key people processes and structures • Involvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system Rating Scale: 1 = A Little; 2 = Just Enough; 3 = Success SCORE =

39. QUESTIONS? ADVICE? FEEDBACK? COMMENTS?

40. CONTACT INFORMATION Primary Presenter: Jill Pfitzenmayer, Ph.D. Title: Senior Vice President, Organizational Quality and Performance Agency: Child & Family Services Address: 24 School Street, Newport, RI 02840 Phone: 401-848-4186 Fax: 401-841-8841 Email: jpfitzenmayer@cfsnewport.org Co-Presenter: David Robinson, Ed.D. Title: Director for the Center for Evaluation and Research with Children and Adolescents (CERCA) Agency: Massachusetts Society for the Prevention of Cruelty to Children Address: 399 Boylston Street, Boston, MA 02116 Phone: 617-587-1594 Fax: 617-587-1582 Email: drobinson@mspcc.org or CERCA@mspcc.org Web: www.mspcc.org