Tools to evaluate policy and environmental changes:

1. Tools to evaluate policy and environmental changes: Presented by: Laura K. Brennan, PhD, MPH Opportunities for growing the field

2. Our Team & Partners • Robert Wood Johnson Foundation • Tracy Orleans, Laura Leviton (and Punam Ohri-Vachaspati) • Active Living By Design • Sarah Strunk, Phil Bors, Rich Bell, Fay Gibson, Joanne Lee, Mary Beth Powell, Tim Schwantes, Risa Wilkerson • Community Partnerships (25 ALbD, 50 Healthy Kids, Healthy Communities) • Washington University Institute for Public Health (St. Louis) • Ross Brownson, Cheryl Carnoske, Christy Hoehner, Peter Hovmand, Timothy Hower • Saint Louis University School of Public Health • Elizabeth Baker, Cheryl Kelly • Transtria LLC • Tammy Behlmann, Sarah Castro, Julie Claus, Peter Holtgrave, Courtney Jones, Allison Kemner, Laura Runnels (many part-time staff and interns)

4. Healthy Kids, Healthy Communities (50 Grantees) HKHC Leading Site Communities Seattle/King County, WA Portland/Multnomah County, OR Houghton, MI Benton County, OR Fitchburg, MA Rochester, NY Milwaukee, WI Buffalo, NY Somerville, MA Kingston, NY Flint, MI Kane County, IL Philadelphia,PA Chicago, IL Omaha, NE Oakland, CA Hamilton County, OH Washington, DC Denver, CO Watsonville/Parajo Valley, CA Kansas City, MO Charleston, WV Central Valley, CA Columbia, MO Louisville, KY Nash/Edgecombe Counties, NC Cuba, NM Baldwin Park, CA Knoxville, TN Moore/Montgomery Counties, NC Rancho Cucamonga, CA Chattanooga, TN Boone/Newton Counties, AR Greenville, SC San Felipe Pueblo, NM Spartanburg, SC Jefferson County, AL Desoto/Marshall/ Tate Counties, MS Phoenix, AZ Milledgeville, GA Grant County, NM Jackson, MS Cook County, GA El Paso, TX New Orleans, LA San Antonio, TX Duval County, FL Houston, TX Lake Worth/Greenacres/ Palm Springs, FL Caguas, PR Healthy Kids, Healthy Communities

5. National Policy/Practice Advisory Group • Don Bishop • Elaine Borton • Leah Ersoylu • Steve Farrar • Harold Goldstein • Dean Grandin • James Krieger • Elizabeth Majestic • Jacqueline Martinez • MalisaMcCreedy • Leslie Mikkelsen • Joyal Mulheron • Thomas Schmid • Marion Standish • Ian Thomas • Mildred Thompson • National Research Advisory Group • Elizabeth Baker • Rachel Ballard-Barbash • Frank Chaloupka • William Dietz • Lawrence Green • Terry Huang • Shiriki Kumanyika • Marc Manley • Robin McKinnon • Shawna Mercer • Meredith Reynolds • Barbara Riley • Eduardo Sanchez • Loel Solomon Our Advisors • National Working • Group • Karen Glanz • Debra Haire-Joshu • Laura Kettel-Khan • Maya Rockeymoore • James Sallis • Janice Sommers • Mary Story • Sarah Strunk • Antronette Yancey

6. Our Projects

7. Connecting Projects • Bridge research/evaluation and policy/practice • Evaluate system, policy and environment change impacts and outcomes • Assess reach, adoption, implementation and sustainability of policy and environment interventions • Accelerate translation of replicable, evidence-based policy and environment interventions • Model the complex pathways from community contextual factors to behaviors and health

8. Why Evaluate? Green LW. Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Family Practice 2008; 1-5.

9. Why Evaluate? Building the scientific evidence

10. Why Evaluate? Shaping, creating policies and practices for the field

11. Why Evaluate? Chances of sustainability are higher for projects that have been evaluated StevensB, Peikes D. When the funding stops: Do grantees of the Local Initiative Funding Partners Program sustain themselves? Evaluation and Program Planning2006;29 (2): 153-161.

12. Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • To identify the changes with the greatest impact, relevance, feasibility and sustainability • To inform local decision-making, document successes & obtain more funding THESE ARE NOT MUTUALLY EXCLUSIVE…

13. Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings

14. Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings OPPORTUNITY: What is a meaningful decrease in BMI? Increase in activity? Decrease in sugar consumption?

15. Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings OPPORTUNITY: Should we use height/weight (BMI)? skinfold thickness? waist circumference? accelerometers? 3-day food recall interviews? surveys?

16. Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings OPPORTUNITY: Prospective cohort designs? Complex time series designs? Natural experiments? What is the intervention population? What is a representative evaluation population?

17. Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings

18. Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings OPPORTUNITY: Partnership or coalition formed? New decision-making body or position created? Policy developed? Policy adopted? Funds appropriated? New/improved structures, facilities or conditions? Policy compliance? Policy enforcement? Use of new facilities? Maintenance of new facilities?

19. Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings OPPORTUNITY: Should we use policy assessment? environmental audits? direct observation? key informant interviews? focus groups? surveys? photovoice or digital storytelling? GIS mapping? web-based tracking systems?

20. Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings OPPORTUNITY: (Perspective of community members) Does the policy or environmental change increase access to healthy foods or opportunities for physical activity? Did community members participate in planning, implementation and evaluation? Can the community sustain the changes over time?

21. Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, business owners)

22. Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, business owners) OPPORTUNITY: Who did the work? For how long? Who provided resources? How much did it cost? What funding was provided/leveraged? What were unanticipated benefits/challenges?

23. Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, business owners) OPPORTUNITY: What policy assessment tools inform policy development? What audit tools can be used to increase awareness? Can digital storytelling be used for advocacy? What surveys help to estimate costs?

24. Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, businesses, schools, residents) OPPORTUNITY: What are the costs? What are the impacts on economic development, academic performance, community safety or air quality? Are changes implemented and enforced equitably throughout the community?

25. Evidence With expanded definitions of evidence, answers to these questions help to bridge the gap between research/evaluation and policy/practice efforts…

26. Shaping the Field

27. Evaluation: Beginning to End

28. Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End

29. Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End

30. Work Plan Planning Goals Objectives Activities Implementation Events Actions Evaluation Outcomes Impacts Processes

31. Work Plan Planning Goals Objectives Activities Implementation Events Actions Evaluation Outcomes Impacts Processes

32. Work Plan Planning Goals Objectives Activities Implementation Events Actions Evaluation Outcomes Impacts Processes

33. Logic Models • What is a logic model? • Description of how activities to be carried out during a project are related to the expected outcomes • Five core components: • Inputs: resources, contributions, investments • Activities: actions, events • Outputs: immediate products (trained staff, meeting attendees) • Outcomes: changes related to your objectives • Impacts: changes related to your goals • Other considerations: • Assumptions, External or Contextual Factors: beliefs about the people involved, interactions and influence of the environment, political context, social determinants of health

42. Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End

43. Evaluation Approach

44. Evaluation Approaches • Understanding system, policy and environment impacts • Similar settings, approaches and strategies • Common tools and/or measures

45. Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End

46. Multi-method

47. Direct Observation Strengths (+) Pre/post comparison (+) Evaluates the impact of physical changes or improvements on behavior Challenges (-) May depend on external factors (e.g., weather, special events) (-) Requires many observations (times of day, days of week)

48. Environmental Audits Strengths (+) Validated tools (+) Pre/post comparison (+) Impact of policies or physical projects on environmental conditions Challenges (-) May not compare across different communities or physical projects (-) May not have facilities or environments to audit at baseline

49. Photos & Videos Strengths (+) Provides visual representation of project impacts (+) Conveys project impacts to diverse audiences Challenges (-) May be expensive depending on equipment and production (-) Requires consent for photo release

50. Policy Analysis CDC Community Health Assessment aNd Group Evaluation (CHANGE) Tool Strengths (+) Identifies policies that hinder or support healthy eating or active living (+) Tools provide potential policies for planning/ implementation W. K. Kellogg Food & Fitness Planning Guide http://www.cdc.gov/healthycommunitiesprogram/tools/change.htm YMCA’s Community Healthy Living Index (CHLI) Challenges (-) May not capture practices (informal policies) (-) May not track policy development, implementation or enforcement http://ww2.wkkf.org/DesktopModules/WKF.00_DmaSupport/ViewDoc.aspx?LanguageID=0&CID=6&ListID=28&ItemID=5000564&fld=PDFFile http://www.ymca.net/communityhealthylivingindex/tools.html