Impact/Outcome Evaluation Training for Local Health Departments

1. Impact/Outcome Evaluation Training for Local Health Departments Webinar Session One April 8, 2013 Presented by: Carolyn Rider & Amanda Linares Research & Evaluation Section

2. The Network IOE Team Carolyn Rider, MA • IOE project coordination • Main point of contact for technical assistance: planning and reporting; data entry templates; evaluation materials and resources; evaluation requirements • Teleconferences • Compendium of Surveys Amanda Linares, MS • IOE data management and analysis • Compilation of submitted IOE data and USDA report generation • Teleconferences • Compendium of Surveys You may be referred to other staff, but your first point of contact is Carolyn.Rider@cdph.ca.gov or IOEval@cdph.ca.gov

3. Today’s Agenda • Evaluation Overview • IOE’s Role in the Network • Overview of IOE Requirements • Resources for Conducting IOE • Required and Optional Surveys

4. Evaluation Overview

5. How it works… • Objective • Identify successful interventions and potential best practices and provide direction for program improvement, refinement, and redirection of effort • Participants • Local Health Departments (LHDs) receiving >$350,000 or choosing Objective 11 as an optional objective • Evaluation Support • Research & Evaluation Section (RES) provides training; quality surveys, evaluation tools and resources; and year-round one-on-one evaluation guidance

6. How it works… • State level analysis of compiled LHD findings enables the Network to assess overall project accomplishment • Individual and compiled LHD findings are shared with USDA annually

7. How it works… • Some aspects of evaluations are standardized • Minimum sample sizes • Standardized survey tools • Pre-test/post-test design • Other evaluation components are designed locally, for example: • Optional survey modules can be added • Control groups

8. Why Evaluate? • Your organization • How can we improve our intervention? • What works? • USDA and other funders • What are we getting for the money we are spending on these programs? • Public • There is a high level of interest in improving diets and physical activity and reducing the prevalence of overweight and obesity

9. Impact Evaluation • Includes a control or comparison group • Strongest designs include randomization • Provides evidence that outcomes are a result of the intervention • Excludes alternative interpretations • Only a few of our IOE projects conduct Impact Evaluation

10. Outcome Evaluation • No control or comparison group • Determines if changes occur in conjunction with an intervention • Cannot exclude alternative explanations, i.e. does not prove that the observed outcomes are attributable to the intervention • Most of our IOE projects are outcome evaluation

11. Impact vs. Outcome Evaluation

12. Questions on … Evaluation Overview?

13. Impact Outcome Evaluation’s Role in the Network

14. IOE’s Role in the Network • What is the history of Impact Outcome Evaluation? • Why do we need to use standardized surveys? • What happens to my data and final report after I turn it in?

15. IOE History • In FFY 04 the Network asked contractors receiving over $500,000 in Federal Share to conduct IOE to proactively demonstrate fiscal responsibility • 12 contractorsparticipated • In FFY 05 the Network lowered the participation threshold to $350,000 • In FFY 07 there was peak participation of 48 contractors • In FFY 12, 40 contractors participated • In FFY 13, 59 contractors, including 17 new LHDs, are required to participate in IOE • In FFY 14, we anticipate all but 10 LHDs will have budgets requiring them to participate

16. Using Standardized Surveys to Evaluate Network Priority Outcomes When individual LHD results are compiled statewide, we can assess the following: • Increased consumption of healthy food – especially fruit and vegetables • Increased consumption of healthy beverages – water and milk • Decreased consumption of less healthy food and sugar-sweetened beverages • Increased physical activity • Increased food security (adults only)

17. Elementary and High School IOE Had Good FV Findings in FFY 12 Elementary/Middle School = 30 contractors; High School = 2 contractors * Times per day

18. Adult FV Findings Were Also Quite Strong Adults= 10 contractors * Cups per day

19. Youth Showed Positive Effects for Drinks and Sweets in FFY 12 Elementary School Elementary/Middle School = 30 contractors; High School = 2 contractors

20. Adults Showed Positive Effects In Addition to FV for FFY 12

21. Youth Showed Gains in Physical Activity Elementary/Middle School = 30 contractors; High School = 2 contractors

22. After the data and final reports are in… • Pre-post data from each set of surveys are combined; then the data are cleaned and basic analysis is performed • A report based on the aggregated data is written for the USDA final report, due near the end of each calendar year • Section 1, or the first 21 questions, of your individual final reports are given to USDA as part of the yearly final report • We do advanced analysis on the combined pre-post dataset • This year, analysis from FFY 12 will be presented at the Society for Nutrition Education and Behavior Conference in August, and hopefully the American Public Health Association Annual Meeting in November

23. Questions on … Impact Outcome Evaluation’s Role in the Network?

24. Requirements and Guidelines for Conducting IOE

25. Local Health Department Scope of Work (SOW) • GOAL 1: • The target population (Supplemental Nutrition Assistance Program-Education (SNAP-Ed)/Nutrition Education and Obesity Prevention (NEOP) participants and those eligible up to 185% Federal Poverty Level (FPL)) is empowered and enabled to select healthy foods and beverages and increase physical activity through nutrition education, social marketing and environmental supports. • Objective 11: • (Impact/Outcome) Annually, conduct and report on Impact Outcome Evaluation (IOE) to assess change in healthy food and beverage consumption and related factors such as perceived benefits of eating healthier foods and beverages, perceived control, self-efficacy, readiness to consume healthier foods and beverages, and perceived diet quality among a group of at least 100 participants.

26. Guidelines for IOE To help LHDs receiving over $350,000 in USDA funds to complete IOE SOW requirements, RES has developed a set of guidelines that provide additional information. This webinar reviews these guidelines. LHDs can also access the guidelines on the IOE website: http://www.cdph.ca.gov/programs/cpns/Pages/ImpactEvaluation.aspx

27. Guidelines for IOE • The Network requires local health departments that receive over $350,000 to conduct impact/outcome evaluation (IOE) using the following guidelines. • The guidelines were developed to generate sound information through rigorous evaluation that can be used to improve nutrition education activities or develop new ones. • Technical assistance is available throughout the year to ensure the guidelines are applied to produce valid evaluation results as well as to fit unique situations.

28. IOE Guidelines Contractors are expected to include an IOE objective in their SOW.

29. IOE Guidelines Contractors are expected to measure change in fruit and vegetable consumption and factors that influence it or beverage consumption and factors that influence it using a pre-test/post-test survey design.

30. IOE Guidelines Contractors are expected to use an age-appropriate tool selected from the following: Children in 3rd-8th grades: Network Youth Survey Adolescents in 9th-12th grades: Network High School Survey or Beverage and Snack Questionnaire 2 (BSQ2) pending approval of Rethink Your Drink (RYD) Adults: Food Behavior Checklist (FBC), Fruit and Vegetable Checklist (FVC), or Adult RYD Survey pending approval of RYD

31. IOE Guidelines • Contractors are expected to evaluate an intervention that has face-to-face contact with the same individuals for at least 30 minutes, on five or more different occasions. • This may include contacts made by sub-contractors or collaborating agencies if it can be demonstrated that the individual has contact at the other site.

32. IOE Guidelines Contractors are expected to submit results for at least 100 matched surveys, i.e., a pre-test (collected before the intervention) and post-test (collected after the intervention) for the same individual.

33. IOE Guidelines Contractors that have previously conducted IOE are expected to do a more rigorous evaluation each year. This may include: • adding a comparison group • increasing sample size • Many other possibilities…

34. IOE Guidelines • Contractors are responsible for collecting, entering, and analyzing their data and are expected to submit data and a report by July 31st of each year. • Data entry templates (DETs) are provided by the Network for all required surveys* to assist with data entry and analysis. *DETs are still under development for the new beverage surveys (BSQ2 and RYD Surveys).

35. IOE Guidelines • Contractors are expected to submit a draft evaluation plan with their final report by July 31st of each year. • This plan should make specific any general language written into the SOW and incorporate findings from the prior year’s evaluation.

36. Questions on … Requirements and Guidelines for Conducting IOE?

37. Resources Available for LHDs Conducting IOE

38. Evaluation Tools & Resources Available to LHDs • Trainings and Webinars • Website • Guidelines for LHDs • Compendium of Surveys, including mandatory and optional surveys • Survey administration protocols • Templates for creating evaluation plans and reports • Data Entry Templates, Codesheets& Instruction FAQ • Planning Teleconferences • One-on-one technical assistance

39. IOE Website http://www.cdph.ca.gov/programs/cpns/Pages/ImpactEvaluation.aspx Planned Website Updates: • IOE Plan and Final Report Templates • Data entry code sheets • 2010-2012 Final IOE Reports • Conference presentations drawn from Network IOE • Other misc research and evaluation resources

40. Survey Administration Protocols • Administration Protocols should be printed from the Compendium of surveys: • Network Youth Survey • Network High School Survey • Instruction Guides are shipped with surveys when they are ordered from the Network: • Food Behavior Checklist • Fruit and Vegetable Checklist • For other surveys, please check the Compendium of Surveys for protocol information.

41. IOE Plan Template Plan templates are provided by RES to assist LHDs to develop a sound evaluation plan. During FFY 2013, this plan was due on 11/30. In subsequent years, it is due 7/31, along with the Final Report.

42. IOE Report Template Final Report templates are also provided for LHDs to summarize intervention activities and evaluation results in a standard format. Final Reports are due annually on July 31, along with supporting data files and the next year’s draft plan. We will discuss this template in depth on Wednesday.

43. Data Entry Templates LHDs use data entry templates (DETs) created for our standard surveys to enter data and produce a basic, standardized analysis This DET has data entered for the Network High School Survey. We will discuss DETs in depth on Wednesday.

44. Technical Assistance: Teleconferences and One-on-one TA • An annual teleconference is scheduled for each LHD after we receive your Final Report and Plan (due July 31) and before beginning the next year’s evaluation. • Who is there? • All necessary staff from your LHD • Any subcontractors essential to your project • Network Program Manager • Network Research & Evaluation representative • What will happen? • Discuss your Final Report from the previous year and your plan for next year; troubleshoot intervention and evaluation • One-on-one TA is available year-round by phone or email. Contact Carolyn Rider.

45. Questions on … Resources Available for LHDs Conducting IOE?

46. Required and Optional Surveys

47. Required and Optional Surveys • Mandatory Survey Instruments • Youth • Network Youth Survey • High School • Network High School Survey or Beverage and Snack Questionnaire 2 (BSQ2) • Adult • Food Behavior Checklist, Fruit and Vegetable Checklist, or Adult Rethink Your Drink Survey • Optional Survey Instruments • NetworkCompendium of Surveys is a comprehensive collection of Network-approved survey instruments

48. Compendium of Surveys • Updated regularly to provide contractors with the best measures we can find to evaluate their programs • Instrument topics include knowledge, self-efficacy, social norms, social support, food security, access to/availability of healthy food and physical activity opportunities, and many more • A new edition is currently on the Network website • Compendium of Surveys

49. Questions on … Required and Optional Surveys?

50. For IOE Training Day 2…