slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
davidr: PowerPoint Presentation
Download Presentation
davidr:

Loading in 2 Seconds...

play fullscreen
1 / 40

davidr: - PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on

davidr:. C-8 Moving Toward Excellence: Integrating Performance and Outcome Measurements. A Data-driven Organizational Quality and Performance System. Jill Pfitzenmayer, Ph.D. Child & Family Services, Newport, RI David Robinson, Ed.D.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'davidr:' - hao


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

davidr:

C-8 Moving Toward Excellence:

Integrating Performance and Outcome Measurements

A Data-driven Organizational Quality and Performance System

Jill Pfitzenmayer, Ph.D.

Child & Family Services, Newport, RI

David Robinson, Ed.D.

Center for Evaluation and Research with Children and Adolescents,

Massachusetts Society for the Prevention of Cruelty to Children, Boston, MA

slide2

Presentation Objectives

1.   Participants will learn how to identify program goals, objectives and specific program activities toward the accomplishment of program mission, vision and values.

2.   Participants will obtain information about establishing an organizational structure that supports critical analysis of program performance and quality improvement.

3.   Participants will identify specific outcome indicators, measurement tools and data collection methods relevant to their program plans.

slide3

Presentation Objectives

4.   Participants will learn how to link staff training, employee retention, program performance and outcome measurement efforts.

5.   Participants will be able to understand when internal outcome assessment can be enhanced by the introduction of an outside evaluator as consultant or primary investigator.

slide4

Agenda

  • Conceptual framework
  • Description of the identified needs and problems
  • Discussion of the methodology and approaches used to address the systems issues,
  • Review of specific agency initiatives and reporting structures developed to enhance organizational quality, and
  • Discussion by an outside evaluator familiar with the agency who was brought in for consultation
definitions

Performance Measurement – “Ongoing monitoring and reporting of program accomplishments, particularly progress towards preestablished goals.”*

    • type or level of program activities (process)
    • direct products and services delivered (outputs)
    • results of products and services (outcomes)
  • Program Evaluation – “Individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working.”*
    • internal or external experts and program managers
    • learn benefits of program or how to improve it

Definitions

* GAO, April 1998

integrating performance measurement and program evaluation a conceptual framework

Integrating Performance Measurement and Program Evaluation: A Conceptual Framework

  • Linked to institutional mission, goals, objectives, strategic plan, process and outcome evaluation
  • SMART - Simple, Minimum paper, Actively used, Revised, and Transferable to new programs
  • Best Practices – continuously updated incorporating new findings, technology and resources
  • Comprehensive - Quantity, Quality, Effort and Effect
performance measurement system

Performance Measurement System

1. Gilbert Performance Engineering Approach

  • Develop a MODEL of process, department, program
  • Define INDICATORS of valued results and expectations
  • Take MEASURES to see how well process is working
  • Implement METHODS of improvement and evaluate results
effective organizational performance measurement systems

Effective Organizational Performance Measurement Systems

  • Commitment – Leadership team committed to measurement
  • Clarity – Degree of clear strategies and metrics
  • Metrics – Indicators measure valuable processes and expectations
  • Alignment – Performance measures aligned with key people processes and structures
  • Involvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system
slide11

Information about CFS

  • Multi-service organization over 136 years old. CEO been in office for 22 years; powerful Board of Directors
  • Primary services:
    • Nine child & adolescent residential sites
    • Two child care centers
    • Family counseling program
    • Home-based program
    • School-based programs
    • Transitional housing program
slide12

Implementation of Continuous Quality Improvement (CQI) Program

  • Minimal COA requirements:
          • Chart review
          • Customer satisfaction
          • Utilization review in select areas
          • Individual outcome data
          • Indication of systematic review of agency
slide13

Implementation Model: Challenges

  • Problem 1: disconnect between accrediting body and agency leadership perceptions of need
    • Problem 2: global discomfort with/ suspicion of “science” and “data”
      • Problem 3: Continuous Quality Improvement is supposed to involve all levels of staff, not be a top-down process
slide14

Steps Required to Develop an Outcome System Year 1:

  • 1. Assemble committee representing key constituents of the agency:
          • Agency leadership
          • Information systems
          • Human Resources
          • Business Office
          • Program staff
          • Parent
          • Consumer
slide15

Steps Required to Develop an Outcome System Year 1:

  • Develop a conceptual model to help staff review program activities and outcomes (e.g., logic models)
  • Map out organizational structure and evaluation activities (e.g., department teams)
  • Identify strategy that makes sense to staff to collect, analyze and use information
slide16

Steps Required to Develop an Outcome System Year 1:

  • Develop protocol and process for all levels of staff participation
  • Develop tools for reporting program activities at least quarterly and identify survey instruments
  • Train staff on recommended model and process
identify a solution focused process
Identify a Solution-Focused Process

PLAN

Decide what is the problem

Brainstorm solutions

DO

Enact solutions

Collect data on how solutions are working

ACT

Revise action plan if needed

Discontinue plan

Continue to monitor

CHECK

Analyze data

Discuss findings

cqi activities related to coa standards
CQI Activities Related to COA Standards

C&FS develops long-term

strategic plan (4 yrs)

(G2.3)

Identified stakeholders

participate in all

appropriate

activities (G2.2)

Each dept. develops

annual plan of goals

and objectives (G2.4)

Internal &

external

audits &

reviews (G2.10,

G2.5)

Chart

Reviews

(G2.6, G2.1)

Customer

satisfaction

surveys

(G2.8, G2.10)

Outcomes

Measures

(G2.7, G2.10)

Quarterly review of each

dept.achievements

by agency CQI Team

(G2.9)

Reassessment,

as necessary (G2.11)

Annual reports

of year-end progress(G2.9)

slide19

Flow of CQI Activities

  • Annual plan
  • Quarterly Monitoring summary
  • Summaries of client/referral satisfaction forms
  • Summaries of incidents that occurred over the quarter
  • Summary of external review activities
  • Review department-specific and aggregated findings
slide20

Year 2: Challenges to CQI System

  • Many staff did not understand logic models
  • Program Directors didn’t understand basic concepts (e.g., difference between benchmark and baseline data)
  • Challenges existed around grasping conceptual material and managing task of putting information on computer!
slide21

Year 2: Challenges to CQI System

  • Some programs never tracked outcome data at all
  • State-funded programs required their own outcome tools that were either cumbersome or lacked validity
  • Staff lacked buy-in, experienced turnover or couldn’t make use of findings
  • Several programs had a hard time identifying individual outcome indicators
slide22

Year 2: Challenges to an Outcome System

  • Mechanically tracking data and outcomes became a challenge—no standardized data base in the agency
  • Changes implemented as a result of outcome data were often not documented
  • Staff were “busy” but couldn’t document real client change (i.e., conflating outputs and outcomes)
slide23

Year 2: Small Successes for an Outcome System

  • Some programs used the process to study and implement real change
    • Residential Network: system for getting paperwork accomplished
    • Family Counseling: saw change in patient population via use of SCL-90-R
    • School-based: outcomes supported pilot program around prevention education
creating organizational change
Creating Organizational Change
  • Year 2: Agency/Board leadership in strategic planning meetings, which required a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis of programs
  • Lack of careful investigation of program outcomes forced leadership to rely more on fiscal and output information than on outcome data
  • Frustration Innovation
creating organizational change25
Continuous Quality Improvement

Reactive

Minimal Board involvement

Driven by COA requirements

Voluntary management participation

Organizational Quality & Performance

Proactive

Board subcommittee

Driven by strategic planning process

Sr VP assigned to job

Creating Organizational Change
coordination of oqp activities
Coordination of OQP Activities

Agency Strategic Plan

Organizational Quality & Performance

Assess performance and outcomes

to align agency goals and program activities

Learning Institute

Link staff needs with training and

learning activities

Employer of Choice Initiatives

Maintain focus on performance

to attract and retain highest

quality staff

moving toward excellence changes in culture and systems
Moving Toward Excellence: Changes in Culture and Systems
  • Staff are now required to identify steps in developing programs reflecting excellence and best practices
  • Management are held accountable for process and program outcomes
  • Systematic data collection and analysis requires new MIS system
  • Agency-wide committees and initiatives brought into oversight process
moving toward excellence changes in culture and systems28
Moving Toward Excellence: Changes in Culture and Systems
  • New OQP committee members include Corporate Communications and Development in order to ensure inter-departmental coordination
  • Methodology for tracking process and outcomes in place
  • OQP SVP meets provides technical assistance in program plan development
moving toward excellence goals of oqp process
Moving Toward Excellence: Goals of OQP Process
  • Department-specific process and outcome data will be collected, analyzed and acted upon
  • Inter-departmental coordination will improve
    • Corporate Com, MIS & Development included in program planning
    • Programs work together on mutual or complimentary projects
    • Inter-program conflicts will be quickly identified and resolved
moving toward excellence goals of oqp process30
Moving Toward Excellence: Goals of OQP Process
  • Communication across departments and up, down chain of command will improve
  • Agency leadership will have improved and speedier methods of determining where there are problems and strengths in programs
  • Agency and program leaders have tools to continuously plan for the future
moving toward excellence how do program staff use oqp system
Moving Toward Excellence: How Do Program Staff Use OQP System?
  • Program Directors use process to bring their vision into program planning
  • Staff are oriented to OQP performance and outcome plans upon hire as a learning tool
  • Program leaders use plans to maintain an eye on program improvement activities
moving toward excellence lessons learned
Moving Toward Excellence: Lessons Learned
  • Appreciate what you bring to the process from your own discipline and theoretical orientation and how they inform your outcome assessment work
  • Ask for help. Use volunteers, experts, etc.
  • Get organized and don’t panic
  • Don’t underestimate the value of communication
  • Maintain a stance of being inquisitive, not judgmental
  • Remind yourself and your staff that the process is developmental
  • Aim for direction, not perfection
slide37

Genuine Unsolicited Testimony from Staff:

“I wanted to share with you my excitement of having completed the baseline ECERS and ITERS for child care today!…With this data we can now create our baseline and use it to establish our movement forward to improved overall quality in our Centers for the children, families and our staff! This is so exciting. Good things are in our future, along with much hard work…I am confident that I will lead the way to our success!”

Patty Shelley, Child Care Program Director

slide38

ASSESSMENT:

Case Study of CQI & Performance Measurement System

  • Commitment – Leadership team committed to measurement
  • Clarity – Degree of clear strategies and metrics
  • Metrics – Indicators measure valuable processes and expectations
  • Alignment – Performance measures aligned with key people processes and structures
  • Involvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system

Rating Scale: 1 = A Little; 2 = Just Enough; 3 = Success

SCORE =

slide39

QUESTIONS?

ADVICE?

FEEDBACK?

COMMENTS?

slide40

CONTACT INFORMATION

Primary Presenter: Jill Pfitzenmayer, Ph.D.

Title: Senior Vice President, Organizational Quality and Performance

Agency: Child & Family Services

Address: 24 School Street, Newport, RI 02840

Phone: 401-848-4186

Fax: 401-841-8841

Email: jpfitzenmayer@cfsnewport.org

Co-Presenter: David Robinson, Ed.D.

Title: Director for the Center for Evaluation and Research with Children and Adolescents (CERCA)

Agency: Massachusetts Society for the Prevention of Cruelty to Children

Address: 399 Boylston Street, Boston, MA 02116

Phone: 617-587-1594

Fax: 617-587-1582

Email: drobinson@mspcc.org or CERCA@mspcc.org

Web: www.mspcc.org