Helping Our Clients Identify and Measure Outcomes

1. Helping Our Clients Identify and Measure Outcomes ESC Brown Bag # 2, April 17 2013 Edward M. Haugh Jr. ESC Consultant

2. Introduction By the end of this session: • Define outcomes • Differentiate outcomes from outputs • Define the logic model • Describe several uses of a logic model • Develop a logic model • Define “indicators” • Develop a logical framework

3. Impact • You will love or at least tolerate outcome-based planning • You will be able to discuss outcomes based planning, monitoring and evaluation with your ESC clients

4. What is an outcome Outcomes are benefits or changes for individuals or populations during or after participating in a program:

5. Outcomes • Often expressed as verbs in past tense • Improved (nutrition, morale) • Increased (in employment) • Reduced (HIV transmission) • Enhanced (knowledge) • Maintained (vaccination coverage)

6. Outcomes • Reduced smoking among teenagers • Changed attitudes towards smoking (by non-smoking teenagers • School readiness among children attending preschool program

7. Other examples • Improved board performance after an ESC engagement • Improved strategic focus after an ESC engagement • Employment after attending a dress-for- success program

8. Improving the Delivery of Prenatal Care and Outcomes of Pregnancy: A Randomized Trial of Nurse Home Visitation

9. Outcomes of Nurse Visit Program • Became aware of more community services • Attended birth classes more frequently • Made more extensive use of the nutrition supplementation program for women, infants and children • Made greater dietary improvements,

10. Outcomes of Nurse Visit Program • Reported that their babies’ fathers became more interested in their pregnancies • Were accompanied to the hospital by a support person during labor more frequently • Reported talking more frequently to family members, friends and service providers about their pregnancies and personal problems • Had fewer kidney problems

11. Outcome Indicators • Outcomes sometimes confused with outcome indicators • Outcome - girls participating in a training program obtain formal employment • Outcome indicators – the # and % of participants employed within 3 months of completing the program

12. Importance of Focusing on Outcomes • Outcomes are what matter to stakeholders • Focusing on outcomes promotes accountability • Government, and funders demanding real tangible results on the ground • Measuring outcomes helps us know if we are we really getting value for our money

13. Outcomes help us • Strengthen existing services • Target effective services for expansion • Identify staff and training needs • Develop and justify budgets • Prepare long-range plans • Focus board on programmatic issues

14. Funder Asked ESC the Following: • What are your measurable outcomes, i.e. number of communities impacted, number of children served?  • Mark each category that applies to your grant measures.

15. Outcomes vs. Outputs • Outcomes are the results • Outputs are the tangible products produced by an intervention • Usually expressed as nouns • Can be counted

16. Outputs • Classes taught • Brochures distributed • Clients served • Contracts completed • Board retreats completed • Strategic plans prepared

17. Many Nonprofits Report Outputs Rather than Outcomes • Main Street Homeless Shelter provided 2,903 nights of safe shelter for 173 men. • 23,661 hot, nutritious meals were served at the Maple Avenue Kitchen. • 100 residents received free health care and 168 residents received free mental health care.

18. What Wrong with Using Outputs • Mosquito bed nets might be used for wedding dresses or fishing nets • Loans repaid might not mean income generated • Job training might not lead to jobs

19. So Why Don’t We Focus on Outcomes • Outputs are tangible • Outcomes are intangible • Its easier to count tangibles than to describe intangibles • Outcomes take longer to produce • Measuring outcomes is challenging • Measuring outcomes can be expensive and time consuming

20. But Tracking Outcomes will Not…… • Explain why an outcome is or is not being achieved • Still need to track inputs, activities, outputs • Prove that the outcome is the result of your program

21. Outcome Model Logic Model Impact Outcomes Outputs Inputs Processes

22. Logic Model Definitions • Inputs are the resources required to undertake activities • Activities (or processes) are what the agency, program or strategy does to fulfill its mission • Outputs – tangible products • Outcomes – changes, benefits • Impact - a longer-term outcome

23. A logic model for a program to distribute HPV Vaccine Medium-term Outcome Processes Long-term Inputs Short-term Outcomes Outputs • Educate consumers • Educate providers • Distribute vaccines • Establish data base • Establish reminder systems • Monitor Coverage • Brochures • Vaccine supplies • Doses of vaccines delivered • Data bases established • Providers monitoring • Reduced incidence of HPV • Reduced number of girls who are infected with HPM • Parents aware of vaccine benefits • Provider awareness • Changed attitudes • Children immunized • Money • Staff time • Facilities • Equipment • Laws • Regulations • Funders requirements • Reduced incidence of cervical cancer • Reduced mortality for cervical cancer Impact Reduced premature death

24. It is important to note the broad environmental context of the intervention: Policy context Political environment Macroeconomic picture Political environment Outcomes Impact Inputs Activities Outputs Environment Other “players” in field Public attitudes Source: Imas and Rist (2009) Political environment

25. How can we use the logic model • ESC Project planning • Strategic Planning • Program Monitoring • Program Evaluation

26. Logic model for an ESC Engagement Short-term Outcomes Medium-term Outcome Impact Inputs Processes Outputs • Contracting • Analysis • Study • Meetings w client • Meetings with stakeholders • Consultant meetings • Progress reports • Draft report • Final report • Board presentation • Staff time • CRM Time • Lead time • Consultant time • Materials • Car and gasoline • Knowledge • Experience • Place to meet ? ? • ? Deliverables

27. Example of a logic model for an ESC Strategic Planning Engagement Outputs Inputs Process Longer-term Outcome Impact Short-term Outcomes Medium-term Outcome Strategic Plan Approved Community benefits from agency’s improved performance Strategic Plan in regular Use Strategic Plan improves agency performance

28. Non-profit Logic Model Short-term Outcomes Inputs Programs Outputs Impact • Money • Staff time • Facilities • Equipment • Laws • Regulations • Funders requirements Program A Output A Outcome 1 Output B Program B Outcome 2 Output C Program C Output D Program D Outcome 3

29. Improved Ability to Fulfill Mission Outcome 4: Improved Public Recognition Outcome 5: New Training Program Developed Outcome 3: Strengthened Volunteer force Outcome 1: Improved Financial Position Outcome 2: New Facility Developed

30. Improved Ability to Fulfill Mission Outcome 1: Improved Financial Position Strategy 1: Capital Campaign Completed Strategy 2: Increased donor base Strategy 3: Client Fees Increased

31. Indicators • Measure the quantity or quality of the outcome

32. Indicators Outcome: Participants in a job training program obtain a job in a timely manner • # (%) of participants employed within 6 weeks • # (%) of participants still employed 3 months after completing the program

33. Indicators Outcome: Homeless people obtain secure housing • # (%) of homeless moved into temporary quarters and stayed for 6 weeks • # (%) of homeless who obtained a job and were still employed after 3 months

34. Indicators Targets • # (%) of participants employed within 6 weeks • # (%) of participants still employed 3 months after completing the program • 60% of participants will gain employment within 6 weeks • 95% of participants who obtained employment will still be employed in 3 months

35. Logical Framework

36. 7. Use the Findings 6. Improve the System What Can Our Clients Do to Measure Outcomes 5. Analyze and Report Findings 4. Try out the Measurement System 3. Prepare to collect data on the Indicators 2. Specify the Indicators for the Outcomes 1. Choose the Outcomes to Measure

37. Two Great Resources • Measuring Outcomes, A Practical Approach, The United Way Foundation • Logic Model Development Guide, WK Kellogg Foundation

38. Thank you for participating in ESC Brownbag No. 2. Look for more Brownbags to come