Dhhs and outcome measurement l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 60

DHHS and Outcome Measurement PowerPoint PPT Presentation


  • 95 Views
  • Uploaded on
  • Presentation posted in: General

DHHS and Outcome Measurement. Wes Albinger, Contract Services Coordinator Milwaukee County Department of Health and Human Services Contract Administration March 16 th , 2006 Presentation will be available at: http://county.milwaukeecounty.org/ContractMgt15483.htm

Download Presentation

DHHS and Outcome Measurement

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Dhhs and outcome measurement l.jpg

DHHS and Outcome Measurement

Wes Albinger, Contract Services Coordinator

Milwaukee County

Department of Health and Human Services

Contract Administration

March 16th, 2006

Presentation will be available at:

http://county.milwaukeecounty.org/ContractMgt15483.htm

Starting Monday, March 20, 2006


Intended outcomes of om initiative l.jpg

Intended “Outcomes” of OM Initiative

  • Initial: Agencies understand DHHS definition and application of OM concepts. Agencies understand the direction DHHS is headed with Outcome Measurement and their role in that process.

  • Intermediate: DHHS articulates long term outcomes for each program area. Agency program activities and initial outcomes are aligned with DHHS long term Outcomes identified for each program.

  • Long Term: Indicators and data collection protocols are established for each program area. OM is part of program and contract requirements.


Dhhs and outcomes why l.jpg

DHHS and Outcomes: WHY?

  • To clearly communicate expectations (goals and objectives) to provider agencies.

  • To establish an objective standard which is consistently applied to measure and report program impact.

  • To provide DHHS and Agencies with a meaningful way to report success to stakeholders (e.g. client, provider, funding agencies, etc.).


Dhhs and outcomes why4 l.jpg

DHHS and Outcomes: WHY?

  • To promote client choice.

  • To increase benefits for clients from contract dollars.

  • To improve program performancevia feedback and program improvement.


Purpose of today s presentation l.jpg

Purpose of Today’s presentation

  • This is a multi-year project that will begin simply with establishment of outcomes and a requirement that agencies report to us how they think they are achieving those outcomes. At this point, DHHS will not be holding programs to a specific set of indicators. In subsequent years, contracts may actually include not only the outcomes, but also the indicators and performance goals.

  • None of this is etched in stone yet -- we have called you together at this juncture specifically to gather your input.


Slide6 l.jpg

FAQ

Q: Is DHHS telling us how to run our program?

A: No. Outcome Measurement focuses more on the impact of the program than the program itself. While the outcome is affected by the program, we are more concerned with the “end” of the program than the “means” to that end. Basically, we are interested in defining where we want programs to go, but not how to get there.

In some instances, external factors prescribe program outcomes or processes. An example would be the new Federal Birth to Three Outcomes which will be required for all BT3 programs. Additionally, the growth in popularity of evidence based program designis putting increasing pressure on programs to adopt proven approaches to service delivery. These will likely play an increasing role in programs.


Slide7 l.jpg

FAQ

Q:What role will Agencies play in this process?

  • A:DHHS will articulate long term outcomes for each program, distilled from estimation of client need, agency practice and capacity, current program requirements, federal, state and other mandates, internal staff, and research. DHHS will solicit information from agencies relating to outcomes, indicators, and data collection methods to help inform this process. Keep in mind that on a program by program basis, the amount of discretionary “slack” in setting outcomes varies greatly.

  • Throughout the process, DHHS will release information for review and comment.


Slide8 l.jpg

FAQ

  • Q: What are the implications of these changes for my Agency in the short term?

  • A: For 2007, we will not be prescribing indicators and performance goals in association with the long-term outcomes. Long term outcomes will be identified for several pilot programs for the FY 2007 RFP. For these programs, you will be asked to describe in detail how your program design and outcomes relate to the long term outcome identified for your program, using successful past performance or a research based justification. Additionally, you will be asked to identify what indicators you are or propose to use to measure the achievement of the outcome, as well as your data collection and or measurement tool(s) (concepts explained shortly).


Faq implications for short term cont d l.jpg

FAQ, implications for short term, cont’d

  • For those programs without DHHS identified long term outcomes in the FY 2007 RFP, you will be asked to describe your current or proposed program design and outcomes, using successful past performance or an evidence based justification. Additionally, you will be asked to identify what indicators you are or propose to use to measure the achievement of the outcome, as well as your data collection and or measurement tool(s).


Faq implications for short term cont d10 l.jpg

FAQ, implications for short term, cont’d

In other words, tell DHHS:

What do you intend to do? (program design)

Why do you intend to do it? How does what you are proposing contribute toward the long term outcome? (rationale/justification)

In what way(s) do you expect clients to benefit? (short, intermediate, and long-term outcomes)

How will you know if you have accomplished this? (indicators)


Slide11 l.jpg

FAQ

  • Q: What are the implications of these changes for my Agency in the long term?

  • A: Indicators and data collection protocols for each program area will be developed through an inclusive process. OM will be part of program and contract requirements.


Slide12 l.jpg

FAQ

  • Q: How will this initiative create work for my agency?

  • A: This initiative is more focused on reporting differently, versus reporting additional information. The required reporting will be part of your semi annual evaluation reports.

  • DHHS will seek to minimize the administrative burden.One way this will be accomplished is by soliciting Agency input to determine current capacity and potential technical assistance needs.


Example agency input l.jpg

Example-Agency input

  • It is worth noting that DHHS has done an extensive review of existing outcome data currently collected to help formulate our thinking and strategy. Many of the examples used in this presentation are actual outcome data already collected; we are looking at the content currently being provided as a starting point.


Agency input l.jpg

Agency Input

You may recall recently completing an Outcome Measurement Survey. These were sent to all agencies with purchase of service contracts. Some notable results, which have already been used to inform our progress on this initiative, are as follows:


Survey responses l.jpg

Survey responses

Agencies have widely interpreted definitions of “Outcome”; as such, there are widely varied applications of Outcome Measurement.


Survey responses16 l.jpg

Survey responses

  • Most indicated an interest in an outcome measurement presentation.

  • Most indicated an interest in a formatted spreadsheet for measuring outcomes.

  • Vast majority are currently reporting outcomes.

  • Approximately half cited fundor requirements as the basis for deciding to report outcomes. It appears that many agencies are not finding an internal benefit to tracking outcomes, but are doing it because they “have” to.


Survey responses17 l.jpg

Survey responses

  • Most already track outcomes for ALL programs.

  • Most common theme that emerged as “concern” was administrative/cost burden, with the second being data reliability and validity.

  • Vast majority have developed a logic model or other visual representation of main programs.

  • Most common methodology employed for measuring outcomes was United Way’s model.


Technical assistance l.jpg

Technical Assistance

  • DHHS will provide technical assistance throughout this process.

  • Let DHHS know what your technical assistance needs are.


On to outcomes l.jpg

On to Outcomes

  • This is intended to give you a sense of the process DHHS is using internally to determine outcomes, and may be helpful to you as you identify how your agency’s program will work toward a specific long term outcome.

  • DHHS will be using a methodology which is already familiar to many of you, and enjoys a broad base of support. It has become most familiar to practitioners through the United Way, but it is used more widely than that.


Some thoughts on terminology l.jpg

Some thoughts onTerminology

Throughout this presentation, I will be using specific terminology. The concept behind the terminology is more important; however, because there is a wide use of terminology and interpretation of terminology among agencies, it is important that:

  • We adopt a common language for discussing Outcome Measurement concepts, and that

  • We use it consistently.


What are outcomes and why measure them l.jpg

What are Outcomes and Why Measure Them?

OUTCOMES-

The intended benefit(s) for participants during or after their involvement with a program. You can also think of Outcomes as the OBJECTIVE of the program.


Assumptions and organizational challenges to implementing om l.jpg

Assumptions and Organizational Challenges to implementing OM

  • Support for OM is based on the assumption that the information produced creates value that exceeds the time and effort to collect it.

  • The challenge, then, is to make sure that the information that OM produces is useful, and that collecting it is administratively as simple as possible.

  • DHHS wants OM information to serve useful purposes for all affected stakeholders.


Assumptions and organizational challenges to implementing om23 l.jpg

Assumptions and Organizational Challenges to implementing OM

  • Concern about OM also comes from the belief that the outcomes measured are not an accurate or complete reflection of the program.

  • Our challenge is to identify outcomes which are appropriate. Your input will help


An outcomes makeover l.jpg

An Outcomes “Makeover”

The Before

  • Served350 people.

  • Hosted10 resume writing seminars.

  • Provided150 hours of individual career counseling.

  • 25 staff attended Outcome Measurement training.

    Notice emphasis on delivery of service, or the program itself.


An outcomes makeover25 l.jpg

An Outcomes “Makeover”

Gee, you sure did a lot, but what happened???

  • Did your clients get jobs?

  • Did your clients learn anything?

  • Did your clients change their lifestyle?

  • In general, how were your clients affected or changed?


An outcomes makeover26 l.jpg

An Outcomes “Makeover”

The After

  • 70% (160 out of 230) of clients over 50 years of age obtained employment .

  • Of the total, 65% were able to maintain employment for at least 1 year.

    Notice emphasis on the benefitor change to the person served. No (or little) emphasis on the actual program. In fact, the program can only be inferred from the outcome statement. These statements reflect the OBJECTIVE of the program.


What exactly is an outcome l.jpg

What Exactly is An Outcome?

Benefits for participants during or after their involvement with a program.

Change in behavior

Change in knowledge or skills

Change in condition


Examples of outcomes l.jpg

Examples of Outcomes

  • Clients decreased emergency room visits.

  • Participants significantly reduced alcohol/drug use.

  • Immigrants were able to secure legal residency and citizenship.

  • Staff demonstrate proficiency with Outcome Measurement concepts.


Outcome or not l.jpg

Outcome or not?

  • Clients increase community activities.

  • Clients participate in skill training programs.

  • Residents feel safer.

  • In 2004, the Advocacy program assisted 75 individuals.

  • Clients reduce emergency room use.


Task 1 choose the outcomes you want to measure l.jpg

Task 1: Choose the Outcomes You Want to Measure

Hints:

  • Be thoughtful!!

  • Define Your Target Population.

  • Engage staff and clients in the discussion.

  • Choose meaningful/compelling outcomes.

  • Write outcomes so “John Doe” will be able to understand.

  • Develop a Logic Model.


Logic model a picture of how a program works l.jpg

Logic Model: A Picture of how a Program Works

Inputs

Activities

Outputs

Outcomes

$

Staff

Outpatient Clinic

Referring agencies

Individuals with mental illness

Outside environment

75 psychiatric sessions

150 drug prescriptions

65 subsidized housing applications

175 crisis calls

80 resume writing/job placement sessions

Psychiatric Services

Pharmacy Services

Nursing Services

Housing Assistance

24 hour crisis hotline

Vocational Services

Reduce level of incarceration of adults with mental illness

Reduce level of hospitalization of adults with mental illness


Levels of outcomes l.jpg

Levels of Outcomes

In most cases, a program attempts to affect not one, but a series of changes for individuals, each building on the previous one. These can be thought of as the stepping stones toward the ultimate, or long term goal of the program. We can define these as:

Initial Outcomes

Intermediate Outcomes

Long term Outcomes


Relationships among levels of outcomes l.jpg

Relationships among levels of outcomes

Levels of outcomes can be thought of in terms of a series of interrelated milestones. The achievement of one sets the condition which increases the chances of achievement of the next.

In other words, IF “a” happens, THEN “b” is more likely to follow.


Relationships among levels of outcomes34 l.jpg

Relationships among levels of outcomes

  • Example:

    IF job seekers learn appropriate job interview skills (initial outcome), THEN they will use appropriate job interview skills in the interview (intermediate outcome). IF they use appropriate interview skills, THEN they will obtain jobs (long-term outcome).


Relationships among levels of outcomes35 l.jpg

Relationships among levels of outcomes

Similarly, it can work backwards: if you know the desired long term outcome for your program, you can ask yourself, “what will make this more likely to occur?”

Example: Intended benefit (Long Term Outcome) for Employment Program: Clients get jobs. What will make this more likely to occur? Answer (one of many possible answers): Clients learn and demonstrate appropriate job seeking skills.

**There should be a theoretical/logical (research or practice supported) link supporting causality between the various outcomes.


Initial outcomes l.jpg

Initial Outcomes

Initial Outcomes-The most immediate changes and most heavily influenced by the program, and are typically expressed in terms of new knowledge, skills, or attitudes learned through the program.


Initial outcomes37 l.jpg

Initial Outcomes

Examples of Initial Outcomes:

Program: Employment

Initial Outcome: Clients learn appropriate job seeking skills.

Program: Adult Day Services

Initial Outcome: Clients increase awareness of community resources.

Program: Outcome Measurement Initiative

Initial Outcome: Staff demonstrate proficiency with Outcome Measurement concepts.


Intermediate outcomes l.jpg

Intermediate Outcomes

Intermediate Outcomes-A link between the program’s initial outcomes and long term outcomes, typically expressed in terms of changes in behavior connected to the new knowledge, attitude, or skill (initial outcome) learned through the program.


Intermediate outcomes39 l.jpg

Intermediate Outcomes

Examples of Intermediate Outcomes:

Program: Employment

Intermediate Outcome: Clients utilize appropriate job seeking skills.

Program: Adult Day Services

Intermediate Outcome: Clients increase utilization of public and private services in their community.

Program: Outcome Measurement Initiative

Intermediate Outcome: Staff apply Outcome Measurement concepts to programs.


Long term outcomes l.jpg

Long Term Outcomes

Long Term Outcomes-The ultimate objective of the program, typically expressed in terms of changes in condition or status. When defining Long Term Outcomes, it is important to select only outcomes which the program can be reasonably expected to influence.


Long term outcomes41 l.jpg

Long Term Outcomes

Examples of Long Term Outcomes

Program: Employment

Long Term Outcome: Clients are hired.

Program: Adult Day Services

Long Term Outcome: Clients generalize acquired skills to other settings.


Important note long term outcomes l.jpg

Important note, Long Term Outcomes

  • Just because outside influences may affect achievement of an outcome, that does not mean the outcome is inappropriate. We can adjust for this by recognizing that outcomes will not be achieved 100% of the time.

  • As an example, consider the effect of the economy (outside influence) on an employment program. The state of the economy cannot rule out the outcome “clients obtain jobs”; rather, the existence of an outside influence means that we should expect to see less than 100% of the outcome achieved.


Important note long term outcomes43 l.jpg

Important note, Long Term Outcomes

  • Long Term Outcomes are of greatest interest to DHHS and will be the primary focus of our efforts to define outcomes for each program.


Example l.jpg

Example

Suppose we identify the long term outcome, “Clients get hired”. Well, we don’t necessarily care if the “means” to that end are the initial outcome “clients learn job seeking skills” or “clients are connected to warm job leads generated by job developers”. Either initial outcome can lead to the long term outcome. Depending on the particular agency’s expertise, service delivery method, or client mix, they may prefer one “means” to the other.


Example45 l.jpg

Example

Another example would be the long term outcome, “clients are appropriate weight.” One program may choose to work toward this end by targeting the initial outcome, “clients are able to identify healthy food” and the intermediate outcome, “clients eat healthy food”, while another program may identify, “clients learn the benefits of exercise” as the initial outcome and “clients exercise at least 2x per week” as the intermediate outcome.


Implications of different means to a common end l.jpg

Implications of Different Means to a Common End

  • Therefore, you can consider the Long Term Outcomes identified for your program to represent the direction we want you to go in. You will be responsible for deciding the best way to get there.

  • In some instances, external factors prescribe program outcomes or processes. An example would be the new Federal Birth to Three Outcomes which will be required for all BT3 programs. Additionally, the growth in popularity of evidence based program design is putting increasing pressure on programs to adopt proven approaches to service delivery. These will likely play an increasing role in programs.


Task 2 specify indicators for your outcomes l.jpg

Task 2: Specify Indicators ForYour Outcomes

Hints:

  • Outcomes and Indicators are not the same thing.

  • Indicators-Measurable approximations of the outcomes you are attempting to achieve. They determine the extent to which the outcome has been met.

  • Indicators look at the outcome and ask, “How do you know?”


Task 2 specify indicators for your outcomes48 l.jpg

Task 2: Specify Indicators ForYour Outcomes

Think about the outcome of the program. Ask yourself:

  • How do I know this Outcome when I see it?

  • What does it look like when this Outcome happens?

  • What are the characteristics of someone who has achieved this Outcome?


Task 2 specify indicators for your outcomes49 l.jpg

Task 2: Specify Indicators ForYour Outcomes

Example:

Program- Recreation.

Outcome- Clients socialize with peers and non-disabled individuals.

Indicator- Number of integrated community activities participated in during the previous month.


Examples of program indicators l.jpg

Examples of Program Indicators


Indicators continued l.jpg

Indicators, continued

Notice how Indicators decrease the ambiguity of determining whether the outcome has been met.


Task 3 prepare to collect data on your indicators l.jpg

Task 3: Prepare to Collect Data on Your Indicators

Hints:

  • Identify data sources for your indicators

    i.e., Records, General Public, Program Participants, Parents

  • Design Data Collection Methods

    i.e., Review of Program Records, Questionnaire, Interview


Things to think about when developing a data collection method l.jpg

Things to Think About when developing a data collection method

  • Collect info that you want to know the answers to (Don’t just do it to satisfy funder requirements!!)

  • Don’t reinvent the wheel (Look for instruments that have already been developed!!)


Task 4 analyze findings l.jpg

Task 4: Analyze Findings

Hints:

  • Make it Easy to Tabulate Your Findings. If possible, computerize!!!

  • Calculate the # and % of clients achieving the outcome.

  • Look at how gender, race/ethnicity, income, etc. influenced achievement of outcomes.


Task 5 use your findings l.jpg

Task 5: Use Your Findings

“If the outcome measurement system is not used for helping the organization improve its services, the time and effort will have been wasted.”

- Outcome Management in NP Orgs , Urban Institute, 2001.


Today s intended training outcomes revisited l.jpg

Today’s intended training “outcomes”, revisited

Initial: Agencies understand DHHS definition and application of OM concepts. Agencies understand the direction DHHS is headed with Outcome Measurement and their role in that process.

Has this been accomplished?


The role for agencies l.jpg

The role for Agencies

  • DHHS will identify long term outcomes for some programs by the release of the 2007 RFP.

  • As mentioned previously, DHHS will define the long term outcomes for each program. Agencies will need to think about how they will work toward that outcome. In reviewing proposals DHHS will look at how logically connected your initial and intermediate outcomes are to the Long Term Outcome defined for the program, as well as how you propose to use indicators and data collection methods.


The role for agencies58 l.jpg

The role for Agencies

  • Agencies can make suggestions for outcomes, indicators, and data collection based on their practices: let DHHS learn from you by showing us what you are currently doing. You may submit this information at any time, but will be asked for it as part of your FY 2007 submission.


The role for agencies59 l.jpg

The role for Agencies

  • DHHS will use your proposed strategies for outcomes, indicators, and data collection methods as a jumping off point as we consider establishing these for each program area.


Resources references l.jpg

Resources, References

  • United Way of America Outcome Measurement Resource Network www.unitedway.org/outcomes

  • Measuring Program Outcomes: A Practical Approach, United Way of America, 1996. (Available through website for $5)

  • Kellogg Foundation Evaluation Handbook & Logic Model Guide www.wkkf.org

  • Urban Institute (Publications by Harry Hatry)

    www.urban.org

  • Fiscal Policy Studies Institute (Mark Friedman)

    www.resultsaccountability.com


  • Login