Understanding Evaluation in Health Promotion: Definitions, Methods, and Outcomes

1. Evaluation 101 Prevention Education Meeting May 29, 2013

2. What isEvaluation?

3. Definitions of Evaluation • Examine and judge carefully. • Determine value or worth. • Process that attempts to determine, systematically and objectively, the relevance, effectiveness, and impact of activities in light of their objectives.

4. Purpose of Evaluation • Use evidence to prove that a program has delivered intended services. • Show that program has made an impact on the health of a certain population. Windsor, Clark, Boyd, & Goodman. Evaluation of Health Promotion, Health Education, & Disease Prevention Programs

5. Purpose of Evaluation • To determine achievement of objectives related to improved health status • To improve program implementation • To provide accountability to funding agencies, community, and other stakeholders McKenzie, Neiger, & Smeltzer. Planning, Implementing, & Evaluating Health Promotion Programs

6. Outputs • The amount of product or service that the program intends to provide • Type of activity, product or service • andtarget audience Participants Clients Agencies Decision makers Customers Public Health care providers • Type of evaluation • process evaluation • Can also include satisfaction with activity/product/service

7. Short Term Outcomes • Immediate effects of activities including changes in Awareness Knowledge Attitudes Beliefs Values Opinions Aspirations Motivation Skills (knowledge) • Type of evaluation • impact evaluation

8. Intermediate Outcomes • Changes in behavior Risk factor reduction Norms Self management Policies Social action Practice Procedures Decision-making Demonstration of skills • Type of evaluation • impact evaluation

9. Long Term Outcomes • Changes in burden of disease, health status, or quality of life • prevalence or incidence • mortality • also social, economic, civic, environmental factors • Type of evaluation • outcome evaluation

10. Data Sourcesfor Evaluation

11. Evaluation Terminology • Process evaluation • measurement of activities during implementation • Impact evaluation • measurement of immediate observable effects of a program including awareness, knowledge, attitudes, skills, and behaviors • Outcome evaluation • measurement of the ultimate goal of a program

12. Process Evaluation Data • Quantitative • Counts • # meetings held, # training sessions conducted, # of people who attended • Descriptive Statistics • mean satisfaction rating for event, percent of people who responded they liked the event • Qualitative • Descriptive information • description of event, date of event, location of event, description of who attended, what information was included during event

13. Process Evaluation Data • Monthly Progress Report • Example: • # professional training sessions/workshops conducted • # professionals trained • # informational materials/brochures distributed • # primary prevention education sessions conducted for students • # students attended primary prevention education sessions • # informational materials/brochures distributed • # sessions conducted with faith-based organizations • descriptions of topics covered during sessions

14. Impact Evaluation Data • Short term outcomes on a logic model • awareness, knowledge, attitude change data • Quantitative • data collected from surveys/questionnaires, written tests • Qualitative • data collected from surveys/questionnaires, interviews, focus groups

15. Impact Evaluation Data • Intermediate term outcomes on a logic model • behavior change data • Quantitative • BRFSS data • Qualitative • interviews, focus groups, observations

16. Impact Evaluation Data • Example: • The results of the pre/post knowledge tests indicted that participants’ knowledge about healthy relationships increased by 500%. • The results indicated that students learned to use negotiation techniques as assessed by post-test. • Responses on the modeling questionnaire indicate that 85% of students would intervene when other students act disrespectfully. • School administrators at 10 schools strengthened their anti-bullying policies.

17. Outcome Evaluation Data • Long term outcomes Changes in health status • Quantitative • YRBS data • BRFSS data • Vital data • Hospital Discharge data • ER or ED data • Qualitative • Interviews, focus groups

18. Outcome Evaluation Data • Example: • 2007 YRBS • 29.9% of high school students had been in a physical fight at least once in the past year • 11.8% of high school students had been hit by their boy/girlfriend in the past year • 9.5% of high school students have been forced to have sexual intercourse against their will • 2008 BRFSS • 12.5% of adult females have been victims of forcible rape • 16.7% of adult females have either been raped or have experienced an attempted rape • 4.7% of adult males have either been raped or have experienced an attempted rape

19. Evaluation Toolsand Resources

20. ABCDE Method of Writing Outcomes • A – Audience (Who will change?) • B – Behavior (What will change?) • C – Condition (By When?) • D – Degree (How Much?) • E – Evidence (How will the change be measured?)

21. Evaluation Activity • Developing a data collection tool

22. General Evaluation Guides • The Program Managers Guide to Evaluation • http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/pmguide_toc.html • Kellogg Foundation Logic Model Development Guide • http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf • Developing a Logic Model: Teaching and Training Guide • http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf • Understanding Evaluation: The Way to Better Prevention Programs • http://www.ed.gov/PDFDocs/handbook.pdf • Kellogg Foundation Evaluation Handbook • http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf

23. Topical Evaluation Guides • Core Performance Measures for Steps to a HealthierUS • http://www.cdc.gov/eval/steps/Steps%20Core%20Performance%20Measures%2007-2007.pdf • Introduction to Program Evaluation for Comprehensive Tobacco Control Programs • http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_evaluation/evaluation_manual/00_pdfs/Evaluation.pdf • Physical Activity Evaluation Handbook • http://www.cdc.gov/nccdphp/dnpa/physical/health_professionals/interventions/handbook.pdf • CDC DASH Evaluation Briefs • http://www.cdc.gov/HealthyYouth/evaluation/resources.htm#4 • Strategic Planning Kit for School Health Programs • http://www.cdc.gov/HealthyYouth/evaluation/pdf/sp_kit/sp_toolkit.pdf • SMART Cards • http://www.cdc.gov/HealthyYouth/evaluation/pdf/SMARTcards.pdf

24. Violence Prevention Evaluation Guides • Sexual and Intimate Partner Violence Prevention Programs Evaluation Guide • Book by CDC, 2007, can be ordered from • http://wwwn.cdc.gov/pubs/ncipc.aspx • Measuring Violence-Related Attitudes, Behaviors, and Influences Among Youths: A Compendium of Assessment Tools, Second Edition • http://www.cdc.gov/ncipc/pub-res/pdf/YV/YV_Compendium.pdf

25. Evaluation Websites • CDC’s Evaluation Framework • http://www.cdc.gov/eval/framework.htm • Community Tool Box • http://ctb.ku.edu/en/ • American Evaluation Association • http://www.eval.org/ • Coalitions Work • http://coalitionswork.com/ • Writing Good Goals and Objectives • http://apps.nccd.cdc.gov/dashoet/writing_good_goals/menu.html • Basic Guide to Program Evaluation • http://www.managementhelp.org/evaluatn/fnl_eval.htm • University of Wisconsin Extension – Program Development and Evaluation • http://www.uwex.edu/ces/pdande/index.html • Kellogg Foundation • http://www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&LanguageID=0