Unit 1: Getting ready

1. Unit 1:Getting ready

2. What comes to mind when you hear the word evaluation?

3. What does evaluation mean to you? On a piece of paper, write down your definition of evaluation.

4. Definition of evaluation Definitions of evaluation may vary by approach and field of practice. We use: Program evaluation is… “the systematiccollection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future program development, and/or increase understanding.” – Patton, 2008 Source: Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage Publications, Inc.

5. In simplest terms, evaluation answers 3 questions: What? So what? Now what?

6. An outcome is… the change that occurs as a result of our activities and investments.

7. A new way of thinking… • FROM: • What are you doing? • What activities do you deliver? • Who participates? How many? • ACTIVITIES • TO: • So what? • What difference are you making? • What results were achieved? • Who benefited? How? • OUTCOMES

8. A metaphor for outcomes: Not how many worms the bird feeds its young, but how well the fledgling flies –United Way, 1999 Source: United Way of America. (n.d.). Outcome measurement: What and Why?: An overview. Retrieved July 31, 2008, from http://liveunited.org/Outcomes/Resources/What/upload/TPsOMWhatandWhy.pdf

9. Create your own “outcome metaphor” Create your own metaphor or drawing that will help you remember the meaning of outcome. Metaphor: a word or phrase denoting one kind of object or idea in place of another

10. What does the term program mean? Program refers to the object of evaluation: direct service interventions, community mobilization efforts, research initiatives, surveillance systems, policy development activities, outbreak investigations, laboratory diagnostics, communication campaigns, infrastructure-building projects, training and educational services, administrative systems. –CDC, 1999 Source: Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health [Electronic version]. Morbidity and Mortality Weekly Report, Vol. 48, No.RR-11, (September 17). Retrieved July 31, 2008, from http://www.cdc.gov/eval/framework.htm

11. Why am I evaluating outcomes? “Don’t accept your dog’s admiration as conclusive proof that you are wonderful.” –Ann Landers

12. What’s in this for you? Benefits of evaluating outcomes: • Program/organizational improvement • Improved accountability • Better ability to describe what a program does – program description for stakeholders • Data for fund raising • Data for public relations • Input for policy decision making • Better planning • Satisfaction in knowing how you are doing

13. Assessing organizational readiness for evaluating outcomes Does your organization have a clearly defined and commonly understood/supported • Mission– what your organization does • Vision– what your organization should look like in the future • Values/beliefs/principles – what you value/ believe – manifested in how you behave

14. Assessing organizational readiness for evaluating outcomes • Are your identified priorities reflected in your goals? • Are the leaders committed to evaluation? • Is there a general commitment to evaluation throughout the organization?

15. Assessing your organization’s readiness… Does your organization have the resources needed to do outcome evaluation? Human • Leadership – committed and involved • Board and staff – understand the purpose • Time – of leadership, staff, and participants Fiscal • Often entails some financial commitment

16. Assessing your organization’s readiness… • Is there a plan and a timeline? • Is there a common framework/language for evaluation? • Do funders/stakeholders expect reporting of outcomes?

17. Assessing your organization’s readiness… Is your organization going to be reporting to a funder? If so, the funder may have a specific format or expectation for reporting outcomes. • Design outcome evaluations to meet the format or expectation. • Remember, mission/vision/values are core to what you do – don’t compromise by “chasing” dollars that don’t “fit” your organization.

18. Assessing your organization’s readiness… • Are staff members interested in outcome evaluation? • Do they have skills to engage in outcome evaluation? • Does the organization support professional development for staff?

19. Assessing your organization’s readiness… • Is evaluation rewarded in your organization? • Are evaluation processes, data, and findings valued? • Are evaluation data and findings used? • Do staff know/see how their evaluations are used?

20. Assessing your organization’s readiness… • Is evaluation data used by the organization to improve programs, guide resource allocations, and support planning? • Is evaluation data used outside the organization to enhance its image, increase funding, and share promising practices?

21. Barriers to evaluating outcomes • Lack of leadership commitment • Limited time and resources, including computer technologies • Limited knowledge and skills in evaluation • Attitude that programs cannot be measured • Fear that evaluation data will hurt programs • Lack of use or misuse of evaluation data

22. Assets in evaluating outcomes • Committed leadership • Resources allocated for outcome measurement • Evaluation expertise exists or can be secured • Evaluations are used • Incentives exist for doing quality evaluation