1 / 91

Partnerships for Evaluating Policy Change

Partnerships for Evaluating Policy Change. Panel Discussion June 13,2006. Why Policy and Environmental Change. Essential Part of Behavior Change. Social-Ecological Model. Policy, Systems, Environment. Community. Institutional/Organizational. Interpersonal. Individual.

Download Presentation

Partnerships for Evaluating Policy Change

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.


Presentation Transcript

  1. Partnerships for Evaluating Policy Change Panel Discussion June 13,2006

  2. Why Policy and Environmental Change • Essential Part of Behavior Change

  3. Social-Ecological Model Policy, Systems, Environment Community Institutional/Organizational Interpersonal Individual

  4. Why Policy and Environmental Change • Essential Part of Behavior Change • Bigger Bang for the Buck

  5. Multilevel Pyramid Model of Stepped Care Interventions Reach Intensity / Cost Highest Lowest Specialty Primary Care Health Systems Community and Neighborhood Media Policies Highest Lowest

  6. Why Policy and Environmental Change • Essential Part of Behavior Change • Bigger Bang for the Buck • Policy and Environmental Changes Stay

  7. Panelists • Amy Roussel, PhD - RTI • Vic Colman, JD – Washington State Department of Health • Shelly Curtis, MPH, RD – The Children’s Alliance • Donna Johnson, PhD, RD – University of Washington • Marilyn Sitaker, MPH - Washington State Department of Health

  8. Evaluation and Policy Change Presented byAmy Roussel, PhD RTI International Presented atAnnual Meeting of the Association of State & Territorial Public Health Nutrition Directors Seattle, WA June 13, 2006 RTI International ■ 3040 Cornwallis Road ■ Research Triangle Park, NC 27709Phone: 919-990-8324 ■ Fax: 919-485-5589 ■ roussel@rti.org ■ www.rti.org RTI International is a trade name of Research Triangle Institute

  9. Session Overview • Define our terms • Demystify evaluation • Discuss types of evaluation • Introduce CDC Framework for Evaluation

  10. What Do We Mean by Evaluation? The systematic investigation of the merit, worth, or significance of program activities or outcomes • Evaluation also explores how and why those activities are occurring • Evaluation builds on the program monitoring activities that your programs currently conduct

  11. What Do We Mean by Policy? A principle, plan, or course of action, as pursued by a government, organization, group, etc. • General term: includes policy, legislation, environmental change, systems change • Occurs at all levels of the socioecologic model

  12. The Socioecologic Model

  13. Some Policy Change Examples • Imposing a 1% sales tax on items listed as “sweet” or “snack” in the USDA National Nutrient Database for Standard Reference • A community building a permanent structure to house a farmers’ market • School systems eliminating deep-fat fryers from all cafeteria kitchens • Worksites subsidizing healthy food choices • Families instituting sit-down dinner policies (& no TV!)

  14. What Have You Done?

  15. More on Evaluation Evaluation helps you • build capacity • manage programs effectively • assess and improve existing efforts • understand reasons for performance • plan and implement new policies or programs • demonstrate the value of your efforts • ensure accountability

  16. Evaluation Types and Questions • Formative evaluation: To develop or improve a policy What kind of policy changes will people respond to? • Process evaluation: To assess process or implementation How [or how well] was the policy implemented? • Outcomes evaluation: To assess outcomes or impact What difference did the policy make?

  17. Start Where You Are • Consider developmental trajectory of policy change • Development – formative evaluation • Implementation – process evaluation • Institutionalization – outcomes evaluation

  18. Consider the Evidence Base • Evidence-based practice • Policy changes known to be effective • Practice-based evidence • Your evaluation will help build the evidence base • Plan your evaluation accordingly

  19. Specifying Goals and Questions • What do you want to know? • What do stakeholders want to know? • What do you intend to do with what you learn?

  20. CDC Framework for Evaluation

  21. 6-Step Model • Engage stakeholders • Describe the program • Focus the design • Gather credible evidence • Justify conclusions • Ensure use and share lessons learned

  22. Step 1: Engage Stakeholders Stakeholders include: • Those interested in the program or policy • Those affected by the policy • Those who are primary users of evaluation

  23. What Stakeholders Can Do Key stakeholder evaluation functions include: • Describing the policy and intended impacts • Selecting evaluation questions and methods • Serving as data sources • Interpreting findings • Disseminating information • Implementing results

  24. Step 2: Describe the Program • Summarize the policy change being evaluated • Establish common definitions and terms • Delineate intended outcomes and other factors that may help or hinder • Describe how the policy fits into the larger picture

  25. Program Descriptions • Often involve logic models – but not necessarily! • Ask yourself: What are we trying to accomplish? • And then what happens? “If you don’t know where you’re going, you’ll end up somewhere else.” -Yogi Berra

  26. Step 3: Focus the Evaluation • Consider the purpose • What questions are you and stakeholders asking? • How will evaluation information be used? • Reality checks • What’s the developmental stage? • What’s the evidence base? • What resources are available for evaluation? • Evaluation questions + policy stage  design

  27. Step 4: Gather Credible Evidence • Qualitative or quantitative? • Primary (new) or secondary (existing) data? • Clear protocols help to ensure rigor Use multiple data sources whenever possible

  28. Step 5: Justify Conclusions • Structured analysis - reproducible • Assess findings against standards • What would constitute success? • Seek stakeholders’ inputs on this early • Interpret the data – what does this tell you?

  29. Step 6: Ensure Use & Share Lessons Learned • Present findings • Discuss findings • Develop action plan • Act on findings Evaluation without action? What’s the point?

  30. Sharing Lessons Learned • Choose your medium – or media • Match method to audience • Allow room for stories, accounts, testimonies • The report is not the end of the evaluation • Solicit feedback from partners, staff, stakeholders • Open dialogue is essential • Share results and brainstorm next steps • Focus on improvement • Let the past inform the future

  31. Acting on Findings • Evaluation ought to be useful and actionable • Participatory planning facilitates action • What needs to be done? • Who will do it? • Who will help? • Turn the plan into action • Keep goal of improvement in mind

  32. Review • Policy – an inclusive term • Evaluation – not so scary any more? • The CDC Framework – a handy tool • Putting it Together – see handout • Key items learned • Action items • SMART objectives

  33. Any Questions?

  34. Victor Colman, JD Senior Policy Advisor Washington State Department of Health Making Policy Choices: Importance of Evaluation June 2006

  35. Learning Objectives • Learn about a state’s approach to improving nutrition and physical activity via the entire spectrum of prevention • Highlight challenges in making policy choices – with a particular focus on the role of evaluation

  36. Nutrition and Physical Activity: The Washington State Story Rare Opportunity Now in Place to Create a “Healthier Washington” • part of Governor’s Health Care agenda • convergence of public, media, public health & policymaker interest in chronic disease prevention, most notably nutrition and physical activity


  38. The Spectrum of Prevention

  39. Our Vision:

  40. Nutrition and Physical Activity: The Washington State Story Coalitions and Partnerships • Washington’s Action For Healthy Kids Team • Access to Healthy Foods Coalition • Washington Coalition for Promoting Physical Activity • Transportation Choices Coalition • Bicycle Alliance of Washington • Breastfeeding Coalition of Washington • Anti-Hunger Nutrition Coalition • WA Association of Local WIC Agencies

  41. Nutrition and Physical Activity: The Washington State Story Importance of Evidence-Based Policy Research • In early 2005, DOH released a “Policy Resource Guide” as a tool for local, state and private interests

  42. Nutrition and Physical Activity: The Washington State Story Importance of Collective Policy Development • Emerging Role of the Nutrition and Physical Activity Policy Leadership Group • Ongoing development of a broad policy platform

  43. Policy Development: Conceptual Framework

  44. Nutrition and Physical Activity: The Washington State Story Importance of Policy Implementation – Recent Bills • SB 5436 – school policy • SB 5186 – physical activity and planning

  45. Nutrition and Physical Activity: The Washington State Story Lessons Learned • Too much focus on change at state capital level • Policy change is, on balance, easier to enact at local and institutional levels • Determining adequate enforcement capacity can be a critical factor in successful policy implementation

  46. Nutrition and Physical Activity: The Washington State Story Challenges • Heightened accountability with government funds • Prevention still held to a higher standard that other kinds of policy choices • Policymakers have short attention spans – prevention-based solutions take time • Government employees can often be confused about their role in policy development

  47. Nutrition and Physical Activity: The Washington State Story Take Home Message When making policy choices, the feasibility of and the need for evaluation should NOT be an afterthought

  48. Nutrition and Physical Activity: The Washington State Story Victor Colman, JD Senior Policy Advisor Division of Community and Family Health, Washington State Department of Health Tel. # 360.236.3721 victor.colman@doh.wa.gov

More Related