measuring success using performance measurement
Download
Skip this Video
Download Presentation
Measuring Success Using Performance Measurement

Loading in 2 Seconds...

play fullscreen
1 / 37

Measuring Success Using Performance Measurement - PowerPoint PPT Presentation


  • 94 Views
  • Uploaded on

Measuring Success Using Performance Measurement. NAEH Conference on Ending Family Homelessness. Michelle Abbenante Brooke Spellman February 8, 2008. What is Performance Measurement?.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Measuring Success Using Performance Measurement' - gus


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
measuring success using performance measurement

Measuring Success Using Performance Measurement

NAEH Conference on Ending Family Homelessness

Michelle Abbenante

Brooke Spellman

February 8, 2008

what is performance measurement
What is Performance Measurement?

Performance measurement is a process that systematically evaluates whether your efforts are making an impacton the clients you are serving or the problem you are targeting.

2008 NAEH Family Conference - Outcomes Workshop

multiple levels of performance measurement
1. Program Level

Program Funding Report, such as HUD APR

Local Service

Provider

Ten-Year Plan Report Card or CoC Application

CoC

2. CoC/System Level

Multiple Levels of Performance Measurement

State-wide Report or Performance Measurement Tool

3. State Level

NAEH Assessment of National Progress or Federal GRPA and PART Reviews

4. National Level

2008 NAEH Family Conference - Outcomes Workshop

why should programs be interested in performance measurement
Why Should Programs be Interested in Performance Measurement?

We are all in the business of helping people, which means we need to…

  • …understand whether current activities are working to achieve intended results.
  • …drive program improvement and share information on effective practices with others.
  • …acknowledge that high-performing programs are more likely to receive funding through competitive funding processes.

2008 NAEH Family Conference - Outcomes Workshop

building blocks of performance measurement
Building Blocks of Performance Measurement
  • Inputs include resources dedicated to, or consumed by, the program—e.g., money, staff and staff time, volunteers and volunteer time, facilities, equipment and supplies.
  • Activities are what the program does with the inputs to fulfill its mission, such as providing shelter, feeding the homeless, or providing job training.
  • Outputs are the direct products of program activities. They usually are presented in terms of the volume of work accomplished—e.g., number of participants served and the number of service engagements.
  • Outcomes are benefits or changes among clients during or after participating in program activities. Outcomes may relate to change in client knowledge, attitudes, values, skills, behaviors, conditions, or other attributes.

2008 NAEH Family Conference - Outcomes Workshop

performance measurement process
Activities

Outreach

Shelters

Case Management

Rent Subsidies &

Services

Inputs

Outputs

$ (CoC and Other)

Programs

Infrastructure

Staff

# Clients Served by Program

Service Linkages

New PSH Units/Subsidies

Vacancy Statistics

Outcomes

30% exited to PH

40% increased income

25% reduction in CH

25% shorter LOS

< recidivism

Performance Measurement Process

Should we adjust how we spend our resources?

How do we document our efforts?

Should we add or change use of resources to expand our impact?

What did our efforts achieve?

2008 NAEH Family Conference - Outcomes Workshop

outputs vs outcomes
Whereas, an outcome is:

Be mindful to distinguish between outputs and outcomes.

Outputs vs. Outcomes
  • An output is:
  • Focused on what the program will do to achieve the outcome.
  • Focused on what the participant will gain from the program.
  • A way to quantify the frequency and intensity of the activity.
  • A way to measure the client-level impact with clear targets and methods for measuring change.
  • Specific to the activity described for the program.
  • Attributable (a result of) to that program.
  • Meaningful and attainable.
  • Feasible and attainable.

If outcomes show the program works… outputs are needed to understand how to replicate results

2008 NAEH Family Conference - Outcomes Workshop

group exercise outcome output or neither
Group Exercise:Outcome, Output, or Neither?

Answers

  • 150 clients received prevention counseling and one-time financial assistance.
  • 90% of persons will obtain employment by completion of program.
  • 75% of program staff will be trained in crisis management techniques.
  • Met 40% (50) of Permanent Supportive Housing goal.
  • 65% of clients with chronic medical condition will improve physical health

Examples

Output

Outcome

Activity

Output

Outcome

2008 NAEH Family Conference - Outcomes Workshop

achieving your outcomes
Achieving Your Outcomes
  • Short-term outcomes: What change will the client experience within a month of his/her involvement in the program? How will you measure this?
  • Intermediate outcomes: What change will the client experience within a year of being involved in the program? How will you measure this?
  • Long-term outcomes: What is the long-term (e.g., 3-year) impact of the program on clients? Has it been sustained? How will it be measured?

Achieving your outcomes can be a progression. Monitoring this progression requires data that are collected at different intervals:

2008 NAEH Family Conference - Outcomes Workshop

example employment program s impact over time
Long-term

Short-term

Example: Employment Program’s Impact Over Time

39% of participants (75% of those who get a job) will retain their jobs for > 12 months.

All of those who get a job (~52 clients) will receive weekly check-up calls and job counseling, as needed.

All of those who complete the training classes (~95 clients) will be referred to jobs and receive job placement counseling.

52% of participants (55% of those who complete the job training class) will obtain full-time employment

93% of participants (97% of people who completed the job training class) will show improved job skills

~95 participants will complete job training classes.

100 people expected to participate

in the program annually

2008 NAEH Family Conference - Outcomes Workshop

framework for converting program goals into outcomes
Step 3

Step 2

Step 1

What do you hope to achieve with this population?

Within the base, how many persons achieved it?

Who is the base for measuring results?

Step 4

Within the base, how many persons achieved it?

Outcome (%)

Who is the base population for measuring results?

Framework for Converting Program Goals into Outcomes
  • How do I convert program goals into measurable outcomes?
  • What do I need to calculate the outcomes?

2008 NAEH Family Conference - Outcomes Workshop

converting program goals into outcomes example
Step 4

20

40

Converting Program Goals into Outcomes: Example

Program Goal: Supporting participants in stable housing at least 6 months

Step 3

Step 2

Step 1

20 people are still in stable housing (6+mo) or exited after being in housing for 6+ mo

Base= Persons who have been enrolled > 6 mo or have exited (n=40 people)

Goal: remain housed > 6 mo

50% remain in stable housing

at least 6 months

2008 NAEH Family Conference - Outcomes Workshop

look out for ambiguous concepts
Look Out for Ambiguous Concepts
  • …obtaining stable housing?
  • …obtaining employment?
  • …increasing income?
  • …accessing services?
  • …becoming more self-sufficient?

Developing and measuring performance outcomes often invites ambiguous concepts into the process.

For example, what do we mean by…

Which data elements and responses will count?

2008 NAEH Family Conference - Outcomes Workshop

hmis data elements are the building blocks of performance measurement
HMIS Data Elements Are the Building Blocks of Performance Measurement

2008 NAEH Family Conference - Outcomes Workshop

wherever possible use hmis to define your concepts
Wherever Possible Use HMIS to Define Your Concepts
  • Emergency shelter

Based on the Destination HMIS data element, we can define stable housing (narrowly) using the following response categories:

  • Apartment or house that you own
  • Transitional housing
  • Staying/living with family
  • Permanent housing
  • Staying/living with friends
  • Substance abuse facility or detox center
  • Hotel or motel voucher paid for without ES voucher
  • Hospital (non-psychiatric)
  • Foster care home or group home
  • Jail, prison or juvenile detention center
  • Place not meant for human habitation
  • Room, apartment, or house that you rent
  • Other
  • Don’t Know
  • Refused

2008 NAEH Family Conference - Outcomes Workshop

group exercise
Group Exercise

Employment Program

The goals of the program are to help unemployed clients obtain employment and help employed clients get “better” jobs. During the past year, the program served 6 (unduplicated) persons:

2008 NAEH Family Conference - Outcomes Workshop

define the base population for each goal
Define the Base Population for Each Goal

Is everyone part of the target population?E.g., Do you expect to calculate an outcome for everyone?

Goal 1: Achieve employment at exit

Goal 2: Obtain “better” employment at exit

2008 NAEH Family Conference - Outcomes Workshop

calculate the outcome for goal 1
Step 4

2

3

Calculate the Outcome for Goal 1

Program Goal 1: Obtain Employment at Exit

Step 3

Step 2

Step 1

All unemployed persons at entry who exited (N= 3)

Achieve

employment

2 persons achieved employment

67% achieved

employment

2008 NAEH Family Conference - Outcomes Workshop

calculate the outcome for goal 2
Step 4

1

2

Calculate the Outcome for Goal 2

Program Goal 2: Improved Employment at Exit

Step 3

Step 2

Step 1

Persons who were employed at entry and exited (N= 2)

Improve employment

1 person increased earnings

50% gained better

employment

2008 NAEH Family Conference - Outcomes Workshop

exercise 2 and the performance measurement process
Inputs

Money: $250,000

Staff: 4 FTEs

1 Facility

Outcomes

67% achieved empl.

50% improved empl.

Exercise 2 and the Performance Measurement Process

Activities

Job Training Classes

Interview Assistance

Job Placement

Services

Outputs

6 enrolled in weekly services

6 employment

assessments

Referred to av. 4jobs each

2008 NAEH Family Conference - Outcomes Workshop

using outcomes to inform future program operations
Step 4

Step 1

Step 2

Step 3

Review outcomes with program managers

Develop action steps and timelines

Regular monitoring

Implement action steps

Using Outcomes to Inform Future Program Operations

2008 NAEH Family Conference - Outcomes Workshop

step 1 reviewing outcomes with managers
Step 1: Reviewing Outcomes with Managers
  • Program director and managers should review outcomes collaboratively to understand what the outcomes are suggesting.
  • Break down the outcomes to understand the underlying forces:

What’s Going On?

  • What are we doing right? What activities contributed to our ability to meet/exceed our benchmarks?
  • Where do we need to improve? What activities fell short of producing the desired outcomes?
  • What else might be contributing to our outcomes? How can we influence or mitigate these external forces to further our positive outcomes?

2008 NAEH Family Conference - Outcomes Workshop

step 2 developing action steps and timelines
Step 2: Developing Action Steps and Timelines
  • Outcomes that were achieved/exceeded: Continue to support the activities that led to our positive performance.
  • Outcomes that were not achieved: Allocate our inputs differently to support different types/levels of activities.
  • Set target dates for reviewing all outcomes—e.g., 3-month intervals.
  • Collaborate with other service providers to “control” the external impacts on the program.

Reinforcing the Good and Adjusting the Bad

2008 NAEH Family Conference - Outcomes Workshop

step 3 implementing the action steps
Step 3: Implementing the Action Steps
  • You can’t implement what you don’t understand: program directors, managers and front-line staff must understand the reasons for making changes in program operations.
  • Information sharing promotes the idea that “we are all in this together.”
  • Information sharing is fluid: program directors, managers and front-line staff can learn from one another; it’s not a one-way (top-down) process.

Getting Buy-In Through Information Sharing

2008 NAEH Family Conference - Outcomes Workshop

step 4 regular monitoring
Step 4: Regular Monitoring
  • Monitor your progress by generating your performance outcomes at different periods of time—e.g., 3-month intervals.
  • Adjust your approach as needed, but usually incrementally.
  • Important to acknowledge that clients’ needs may shift, and thus program goals and approach may also need to shift.

It’s Easier to Adjust Program Operations Incrementally than Wholesale

2008 NAEH Family Conference - Outcomes Workshop

comparing program results
Comparing Program Results
  • You can compare results from one program to another to see which programs are working best and which are working least well
    • With limited dollars, you want to fund the programs that are most effective.
    • You can use program results to identify best practice programs and those that need TA
    • You can use results from multiple programs to help set a community expectation or standard of performance

2008 NAEH Family Conference - Outcomes Workshop

case study how washington d c uses program results
Case Study: How Washington, D.C. Uses Program Results
  • Outputs/Efficiency Measures
    • Clients Served
    • Chronically Homeless Served
    • Occupancy: the rate at which program was used
  • Interim Outcome Measures
    • Permanent Housing
      • Positive client destinations at exit (TH programs)
      • Retain clients for 6+ months (PSH programs)
    • Income: the amount of income or sources obtained
    • Self-Sufficiency: change in substance use, education, mental illness or employment
  • Measures apply differently to each program type and are supplemented with qualitative data for ranking purposes

2008 NAEH Family Conference - Outcomes Workshop

slide28
Washington, DCFY 07 DHS Performance Measures

*Self Sufficiency Temporary, Transitional and Permanent Supportive Housing Programs required to submit a Self Sufficiency Indicator will have to choose from Substance Abuse, Education, Mental Illness Assistance or Employment.

2008 NAEH Family Conference - Outcomes Workshop

apples to apples risk adjustment
Apples to Apples: Risk Adjustment
  • Problem: Comparing program results can encourage programs to “cream” to ensure strong results
  • Solution: Risk adjustment allows you to account for differences in client populations when comparing results across programs

Results can be adjusted on the basis of…

      • Client characteristics, such as demographics, family size, disability
      • Client history, such as past eviction, criminal background,
      • Client functionality or level of engagement/commitment to change

2008 NAEH Family Conference - Outcomes Workshop

risk adjustment requires expertise
Risk Adjustment Requires Expertise
  • To adjust for client differences, programs need to collect consistent data on clients to use during analysis of program results
    • Agree on these standards beforehand
  • Develop an analysis plan for how you intend to adjust for client differences
    • Engage a researcher to help develop the plan
  • Even if you don’t formally adjust results, acknowledge that different programs may have different outcome expectations based on differences in clients targeted and/or served

2008 NAEH Family Conference - Outcomes Workshop

simplified illustration of dissecting client outcomes on increased earned income
CoC Outcome

Results By

Population

All Clients (n=300)

All Clients (n=100)

61% (61 clients)

23% (70 clients)

Disabled Clients

Disabled Clients

Disabled Clients

12%

(30 of 250 persons)

10%

(1 of 10 persons)

12%

(31 of 260 persons)

Non-disabled Clients

Non-disabled Clients

Non-disabled Clients

67%

(60 of 90 persons)

80%

(40 of 50 persons)

71%

(100 of 140 persons)

Simplified Illustration of Dissecting Client Outcomes on Increased Earned Income

Program A

Program B

By establishing targets, programs can be compared against CoC expectations in the future to determine if program performance is higher or lower than expected.

2008 NAEH Family Conference - Outcomes Workshop

system performance measurement
System Performance Measurement
  • Are your actions achieving your intended goals at the system level?
    • Does the system work?
    • If yes, what makes it work?
    • If no, what part doesn’t work, and how do you fix it to make it work?
  • Note that you may have system goals that only relate to certain types of clients or parts of the system (e.g., different goals for severely disabled persons)

2008 NAEH Family Conference - Outcomes Workshop

sample impact measures
Sample Impact Measures
  • Incidence of homelessness - Is homelessness declining?
  • Incidence of street or CH - Is street or chronic homelessness declining?
  • Length of stay in system, across all homeless programs - Do people stay homeless for shorter periods of time?
  • Prevention – Are fewer people experiencing homelessness for the first-time?
  • Rates of Recidivism – Are repeat occurrences of homelessness avoided or declining?

Cross-tabulate results by core characteristics to understand if/how results vary for different subpopulations

2008 NAEH Family Conference - Outcomes Workshop

steps to calculate system length of stay
Step 3

Step 1

Calculate LOS for each Stay

De-duplicate clients across programs

Step 2

Step 4

Create table with all Entry/Exit Dates by Client

Consolidate sequential stays into single episode (gaps < 30 days = same episode)

114

Step 5

Calculate mean (168 days), low (81 days), high (309 days)

Steps to Calculate System Length of Stay

LOS

229281309

Entry Exit Date Date

5/8/06 5/30/076/1/07 9/01/07 3/1/07 5/21/07 2/1/06 12/7/06

2008 NAEH Family Conference - Outcomes Workshop

some notes of caution
Some notes of caution…
  • There is more to performance measurement than conducting the analysis
    • Educate, train, obtain buy-in
  • Be careful about how you interpret and use the data
    • Jump in, but don’t be careless in how you use the results
    • Look at the results within the context of all the outputs, interim measures and impact measures to validate the interpretation that’s being made
    • Vet the results before publicly releasing anything
    • Appropriately caveat the limitations of the data and analysis

2008 NAEH Family Conference - Outcomes Workshop

summary of system performance measurement
Activities

Adjust type and intensity of activities based on outcomes; track if improves

Inputs

Outputs

Use program-level &

system-wide results

to adjust use of

resources

Document the level of effort provided

Use to ensure activities

delivered efficiently

Outcomes

Interim Outcomes signal client success;

Impact outcomes track progress to

goals

Summary of System Performance Measurement

2008 NAEH Family Conference - Outcomes Workshop

questions

Questions?

Contact us for more information or assistance:

Michelle Abbenante, [email protected]

Brooke Spellman, [email protected]

ad