Developing Public Health Performance Measures: Benefits and Challenges

1. Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October 2001

2. About Us • Maricopa County Department of Public Health in Phoenix, AZ • 440 employees • 47 distinct programs (from childhood immunizations to HIV education) • 85% grant funded

3. Developing Measures

4. History • Pre-1998: Collected some metrics (mostly grants) • 1998-99: Dept. leaders request collection of measures for all programs • 2000: Countywide initiative • 2000-01: Complete “family” of measures for all 47 programs

5. Developing Measures • One employee dedicated to process coordination (about ½ time) • Trained managers/supervisors on: • Types of pms (e.g. output, outcome) • Proper collection methods • Proper documentation

6. Developing Measures • Supervisors sat down with staff to determine goals • When needed, coordinator worked individually with programs • Biggest challenge: the psychology…

7. Stages of PM Process* • Denial: “This is ‘flavor of the month’. If I ignore it, it will go away shortly.” • Anger: “I haven’t got time for this!” • Panic/Freeze: PMA (Performance Measure Anxiety) • Depression: “I can’t wait to retire.” • Acceptance: “We have to do this.” * Apologies to Elisabeth Kubler-Ross

8. Two Sales Options Cheerleader Model: Telling employees how super-fantastic these performance measures will be!!! Empathy (I Feel Your Pain) Model: Acknowledging hard work that will be required by employees. Helping them through difficulties.

9. Collection and Maintenance • Maintain PMs in central document/ electronic location w/gatekeeper • Important to keep narrative explaining significant shifts in numbers • Updated semi-annually, now quarterly

10. Example: Immunizations • Result: Coverage level - % of clients receiving entire series • Output: # of immunizations, # of clients, # of clients 0-2 years old • Efficiency: Cost perimmunization

11. Example: Nutrition Education • Result: Increase in knowledge, attitudes, behavioral intent from pre- to post-test • Output: # of students • Efficiency: Cost per student

12. Using and Abusing Measures

13. Philosophy Set Goals Funding Decisions (“Accountability”) Run Program Measure and Analyze Program Results

14. Benefits • Long term resource planning • Staff • Budget • A goal to “shoot for”, rally around • Sticking with priorities – are we using our energies for our top priority?

15. Do we really measure performance?

16. Performance Measurement Experiment Maricopa County County X Goal for Fiscal Year 01: Goal for Fiscal Year 01: 25 25 Actual Performance: Actual Performance:

17. Performance Measurement Experiment At this point in the presentation, two people – representing each of the counties – must fill a cup with marbles. Maricopa County uses a spoon and County X has to use chopsticks. The result is that County X scores lower on its performance measure. Without knowing that County X had chopsticks, we might conclude that it exhibits “poor performance.” Maricopa County County X Goal for Fiscal Year 01: Goal for Fiscal Year 01: 25 25 Actual Performance: Actual Performance:

18. The “Chopstick Effect” • Funding • Staff shortages • Change in grant requirements • Marlboro has big ad/promotion • And so on…

19. Issues • May not measure program effort or potential effectiveness • “Widget Factory” model often not applicable in PH • Cost of collecting data some-times not worth it • Goal setting as art

20. Alternatives • Process evaluation • “Up front” research (usually from Academia) • Statistical modeling • Standard research practices:Test/Control, etc. • National Public Health Performance Standards (self-assessment)

21. Conclusions • You can develop measures for your organization… • But don’t forget the psychological component. • Use performance measures wisely – better for planning than evaluating. • Let’s find other ways to accurately determine success (for true “accountability”).

22. Information • Mare SchumacherMaricopa County Department of PH1845 E. Roosevelt, Phoenix, AZ 85006 • Mareschumacher@mail.maricopa.gov • Resource: “Taking Stock of Performance Measurement: Information Resources for Public Managers”, Government Finance Review, April 2001 • Resource: “Performance Measures Basics” (presentation for program staff). Send e-mail to Mare Schumacher for copy.

23. The following is a presentation I used to educate program staff on performance measures at our Department. (It was not presented at the APHA.) Feel free to use it or alter it for your own purposes.I ask nothing in return except that you think of me for a moment every time you use it… Mare Schumacher Department of Public Health Maricopa County, Arizona (Phoenix)

24. Performance Measures Presentation to Public Health Strategic Planning Team March 1999

25. The Past • The night before due, make up #s to put in as performance measures • No one looks at them • No one checks to see if you measured • No one thinks about it for 11 months and 27 days • Next year, repeat process

26. The Future • County Administration will be looking closely at performance measures • Depts./Programs will be judged on performance measures • Budget may be tied to performance measures

27. The night before due, make up #s Live with consequences Come up with performance measures that are meaningful to County and you Making the Best of It 1. 2. • We’ve got two choices:

28. Suggested Process May be other options...

29. Meet with Staff • In a meeting with all or many staff • Helps set realistic goals • Employees know what is expected of them • “What gets measured gets done.” • -- Rose Howe, December 1998

30. Decide Program Mission • Why do we exist? • What contribution to we make to the people of MC? • NOT What do we do?

31. What we do: Have meetings Contact coalition members Prepare reports Get trained in how to build teams Why we exist To create a comprehensive HIV prevention plan that will reduce the spread of HIV Example: HIV Planning

32. Just talk this • We’re not measuring what we do • We are measuring our contribution to the public health • Doesn’t mean what we do is not important, just not the thing we’re measuring

33. Output measurement: Activities Things we do Examples: Number of applications processed Number of clients Number of immunizations Number of coalition meetings Create Output Measurement

34. Example: Immunizations* • Conducted 7500 immunizations in 97/98, etc. • Expect to administer 9500 in 99/00 • Reasons: Population growing by 5%, will have increase in staff in 99/00 * All numbers made up -- demonstration purposes only

35. Efficiency measurement: Cost per output Resources used to produce the activity Examples: Cost per client Time spent per client Cost per client compared to certain standard(s) Create Efficiency Measurement

36. Example: Immunizations • Clients per hour = 4 in 97/98 • Expect 5 by 2000/01 due to new methods • Cost per client = $1.22 • Expect cost = $1.45 by 00/01 due to increase cost of vaccine.

37. Impact (outcome) measurement: Indicate the impact of service on the community Should flow directly from mission Examples: Percentage of clients rehabilitated Reduction in incidence of disease Create Impact (Outcome) Measurement

38. Example: Immunizations • 90% coverage of school-aged children by 00/01 • 57% coverage of clients at public health immunization sites • Possibly: 400 cases of measles avoided

39. All Measures • What data will we use to measure it? • If none available now, how will we get it?

40. Goals (Future Years) • Make them specific • Make them realistic - how much control do we have over that measure? • What are the reasons for expecting the goal to be higher, lower, etc. • Document why you picked that goal? How did you calculate it?

41. Develop measures with staff Draft meeting with John/Mare Final draft meeting Final measures due End of March/April End of April (sign-up sheet) May 11 SP Meeting June 1 Timing