The Path to Effective Tobacco Counter-Marketing C Rebecca Murphy-Hoefer, PhD, MPH May 28, 2008 Maple Hill Farm Hal

2. Objectives Present CDC Framework for Program Evaluation as a framework or �Roadmap� Highlight Program Planning, Managing and Implementation, and Evaluation Explore Program Planning, Best Practces and Lessons Learned

3. Why Invest in Media Campaign Evaluation? Demonstrate accountability to stakeholders Media Campaigns most visible and $$$ Measure program achievement Are KABB changing? Manage program resources Managing calls to Quitline Document and improve program operations Feedback on ads, placement, funding The evaluation of the counter-marketing efforts should be coordinated with the evaluation of the entire tobacco control program Why invest in program evaluation? The answer is simple and clear... evaluation allows us to measure program achievement on several levels. As a result of evaluation efforts we are able to demonstrate accountability to funding sources, stakeholders, and the communities we serve. We are better able to manage resources, including the ability to determine if existing resources are being used most effectively and examine what resources are still needed. Evaluation informs program planning toward improved program operations. Why invest in program evaluation? The answer is simple and clear... evaluation allows us to measure program achievement on several levels. As a result of evaluation efforts we are able to demonstrate accountability to funding sources, stakeholders, and the communities we serve. We are better able to manage resources, including the ability to determine if existing resources are being used most effectively and examine what resources are still needed. Evaluation informs program planning toward improved program operations.

4. CDC�s Framework for Program Evaluation 

6. CDC�s Designing and Implementing an Effective Tobacco Counter-Marketing Campaign

7. Framework for Program Evaluation in Public Health

8. Outcome Evaluation Results of campaign Changes in attitudes or beliefs about tobacco Changes in intentions Changes in behaviors Changes in the environment Changes in disease trend Activity Activity: Shaw ads and rate based on persuasion scale (attn, think, talk)Activity: Shaw ads and rate based on persuasion scale (attn, think, talk)

9. CDC Framework for Program Evaluation

10. Commonly Used Indicators Participation rates Communities reached Attitudes Individual behavior (prevalence) Community norms Policies Health status http://www.cdc.gov/tobacco/Indicators/KeyIndicators.htm Primary and secondary data

11. Framework for Program Evaluation in Public Health

12. Justify Conclusions Analyze and Interpret Types- Descriptive, comparative statistics, segments, time, region, level of awareness of campaign, internal, external Attribution- Campaign accounts for difference Changes overtime, exposure effects , regional effects Multivariate Analyses �Effects of multiple IVs (timing of ads, change in awareness, attitudes) on DVs (change in behavior), control for age, sex, race, ethnicity Measure attitudes, beliefs and behaviors expected to change those not expected to change Longitudinal study-Changes over time Field study-Compare communities (etc.) Web-based

13. Framework for Program Evaluation in Public Health

14. Ensure Use Support annual and long-range planning Focus attention on issues important to program Promote your program Identify partners for future collaborations Enhance the public image of your program Retain or increase funding Provide direction for program staff

15. Checklist for an Effective Report Include an executive summary Describe the stakeholders and involvement Describe features of the program, include the logic model Outline key evaluation questions Include a description of the methods, methodological strengths and weaknesses Present results and conclusions into context (what is reasonable at this point and how the results should be interpreted) Translate findings into recommendations Minimize technical jargon Provide detailed information in appendices Use examples, illustrations, graphics, and stories Involve stakeholders in preparation of the report Consider how the findings might affect others Develop additional communication products suited to a variety of audiences, for sharing the results

18. CDC�s Framework for Program Evaluation 

19. Involve Stakeholders�Why? Ensure that media evaluation designed to answer questions important to stakeholders Increase likelihood of continued support Build wider competency in evaluation Increase possibility evaluation findings used Clear understanding of limitations Clarify goals, objectives and limitations with stakeholders Making broad conclusions are difficult Involving stakeholders is important for several reasons: Creating a collaborative environment will increase buy-in or support for your program and help to ensure that the evaluation is used -- keep in mind the purposes of evaluation -- if the information is not used, your time was not well spent. Involving stakeholders also aids in the process of prioritizing evaluation questions -- we need to address questions important to the people we serve.Involving stakeholders is important for several reasons: Creating a collaborative environment will increase buy-in or support for your program and help to ensure that the evaluation is used -- keep in mind the purposes of evaluation -- if the information is not used, your time was not well spent. Involving stakeholders also aids in the process of prioritizing evaluation questions -- we need to address questions important to the people we serve.

20. Involve Stakeholders�Why? Reduces suspicion and fear Encourages differing perspectives and many voices Increases awareness and commitment Increases the possibility of reaching objectives Feel ownership, defend the program Avoid ignored, criticized, or resisted. Will support... From David Cowling, PhD CA DOH: External stakeholders Administration in Department of Public Health Evaluation Advisory Committee Stakeholders in California want input into media campaign not the evaluation of it Evaluation & media unit and internal decision-makers are interestedFrom David Cowling, PhD CA DOH: External stakeholders Administration in Department of Public Health Evaluation Advisory Committee Stakeholders in California want input into media campaign not the evaluation of it Evaluation & media unit and internal decision-makers are interested

21. Potential Stakeholders Program staff, Campaign�s operation Local and regional coalitions Community leaders, members, and grantees Local, state, and national partners Program funders Target audience The health department and commissioner Law enforcement Schools and educational groups The medical community Community-based organizations, advocacy groups Elected officials

22. Checklist�Engage Stakeholders Target key stakeholders for participation Identify key areas for stakeholder input Balance inclusivity, individuals, and orgs Create a plan for strategic involvement Bring stakeholders together regularly Understand and reflect stakeholder values, increase credibility Establish a method of communication Document the need for the program Examples: Health problem and consequences Magnitude of problem overall and by subgroups Determinants of problem(s) Target groups Changes or trends taking place State program goals and objectives (SMART) List program activities Document program resources Note the stage of development Explain the program context Examples: politics, funding, history of tobacco control in state/community, support or lack of support, pertinent social, cultural, or environmental conditions, advertising, lawsuits/legislation/taxes/policy Document the need for the program Examples: Health problem and consequences Magnitude of problem overall and by subgroups Determinants of problem(s) Target groups Changes or trends taking place State program goals and objectives (SMART) List program activities Document program resources Note the stage of development Explain the program context Examples: politics, funding, history of tobacco control in state/community, support or lack of support, pertinent social, cultural, or environmental conditions, advertising, lawsuits/legislation/taxes/policy

23. Establish a Media Evaluation Team Budgeting Developing and communicating program objectives and the logic model Managing evaluating contracts Coordinating evaluation activities between program staff and the evaluation team Incorporating evaluation findings into program planning and revision Technical expertise to design and implement specific tasks Health departments with personnel with technical expertise can be a part of the evaluation team, but should involve outside evaluation experts (neutral and objective)

25. CDC�s Designing and Implementing an Effective Tobacco Counter-Marketing Campaign

26. Describe the Program Stage of development Context Need Resources Goals (CDC) SMART objectives Activities Expected effects Logic model This includes: Characterizing the needs addressed by the program Listing expected effects of your program Illustrating the relationships between program activities and expected changes Assessing stage of development or maturity of your program Making visible the context within which the program operates Doing this may enable you to identify gaps or missing links between program activities and desired results. This includes: Characterizing the needs addressed by the program Listing expected effects of your program Illustrating the relationships between program activities and expected changes Assessing stage of development or maturity of your program Making visible the context within which the program operates Doing this may enable you to identify gaps or missing links between program activities and desired results.

27. Goals & Objectives Goals Prevent the initiation of tobacco use among young people Promote quitting among young people and adults Eliminate exposure to secondhand smoke Identify and eliminate the disparities related to tobacco use and its effects among different population groups Objectives Select a limited set of objectives to focus on the most important results feasible to obtain

28. Strong Program Objectives Are Performance objectives are most useful if they are : By Specific identify a specific event or action that will take place. By Measurable identify the amount of change to be achieved By Achievable should be able to be reached in your designated time frame By Relevant a logical extension of the larger goal area Time-bound provide time constraints for achievement. Performance objectives are most useful if they are : By Specific identify a specific event or action that will take place. By Measurable identify the amount of change to be achieved By Achievable should be able to be reached in your designated time frame By Relevant a logical extension of the larger goal area Time-bound provide time constraints for achievement.

29. Strong Program Objectives Are Good objectives are SMART � Specific, Measurable, Achievable, Relevant and Time-bound. Let�s look at how the objective on the right fits all of these criteria. Clearly defined objectives are critical to program evaluation because these objectives identify the targets you will use to measure your program�s success. But we need to realize that each setting is unique in its priorities. This results in a variety of program designs based upon available resources, political context, and identified needs. Therefore, it becomes vital that we spend time thinking about and developing specific program objectives. In order to measure our progress toward program goals, objectives need to be written very clearly. Well written and clearly defined objectives are SMART: Regarding aided and unaided, I've reviewed a paper by Farrelly et. al. ( 2002) American Journal of Public health that basically showed their campaign unaided was at 22% awareness of TRUTH and aided was 75%; compared to the tobacco industry-Think. Don't Smoke. was 3.2% unaided and 65% aided.� Good objectives are SMART � Specific, Measurable, Achievable, Relevant and Time-bound. Let�s look at how the objective on the right fits all of these criteria. Clearly defined objectives are critical to program evaluation because these objectives identify the targets you will use to measure your program�s success. But we need to realize that each setting is unique in its priorities. This results in a variety of program designs based upon available resources, political context, and identified needs. Therefore, it becomes vital that we spend time thinking about and developing specific program objectives. In order to measure our progress toward program goals, objectives need to be written very clearly. Well written and clearly defined objectives are SMART: Regarding aided and unaided, I've reviewed a paper by Farrelly et. al. ( 2002) American Journal of Public health that basically showed their campaign unaided was at 22% awareness of TRUTH and aided was 75%; compared to the tobacco industry-Think. Don't Smoke. was 3.2% unaided and 65% aided.�

31. Waterford, ME;, Intersection of Route 5 & Route 35

32. Generic Program Logic Model The basic components of a logic model include inputs, activities, outputs, and outcomes. READ DEFINITIONS As we�re identifying each of these components, we need to take into account the context in which the program operates. This is important because we are going to break outcomes down into short, intermediate, and long term. CLICK What is a logic model? -Disciplined way of mapping a program -Platform for discussion -Multi-purpose tool -Presentation of links in a chain of reasoning The basic components of a logic model include inputs, activities, outputs, and outcomes. READ DEFINITIONS As we�re identifying each of these components, we need to take into account the context in which the program operates. This is important because we are going to break outcomes down into short, intermediate, and long term. CLICK What is a logic model? -Disciplined way of mapping a program -Platform for discussion -Multi-purpose tool -Presentation of links in a chain of reasoning

33. Here is an example of a very simple logic model. We start with inputs, which are resources dedicated to or consumed by the program. Using the inputs, we accomplish activities, or what the program does with the inputs to fulfill its mission. This could be types of programming or programming strategies. The activities lead to outputs, the direct product of program activities. For example, outputs could be number of sessions conducted, hours of service delivered, or number of participants served. Finally, the outputs lead to longer term outcomes, or the actual benefits for participants during and after program activities. This could be increased knowledge, changed attitudes, or changes in behavior and health effects. If we choose programmatic activities that are science based and shown to have an impact, we can expect that our activities and outputs will truly lead to a change in outcomes. Here is an example of a very simple logic model. We start with inputs, which are resources dedicated to or consumed by the program. Using the inputs, we accomplish activities, or what the program does with the inputs to fulfill its mission. This could be types of programming or programming strategies. The activities lead to outputs, the direct product of program activities. For example, outputs could be number of sessions conducted, hours of service delivered, or number of participants served. Finally, the outputs lead to longer term outcomes, or the actual benefits for participants during and after program activities. This could be increased knowledge, changed attitudes, or changes in behavior and health effects. If we choose programmatic activities that are science based and shown to have an impact, we can expect that our activities and outputs will truly lead to a change in outcomes.

35. Regarding aided and unaided, I've reviewed a paper by Farrelly et. al. ( 2002) American Journal of Public health that basically showed their campaign unaided was at 22% awareness of TRUTH and aided was 75%; compared to the tobacco industry-Think. Don't Smoke. was 3.2% unaided and 65% aided.� Regarding aided and unaided, I've reviewed a paper by Farrelly et. al. ( 2002) American Journal of Public health that basically showed their campaign unaided was at 22% awareness of TRUTH and aided was 75%; compared to the tobacco industry-Think. Don't Smoke. was 3.2% unaided and 65% aided.�

38. Special Notes While logic models list behavioral outcomes, results occur through a combination of interventions. Example: a media literacy program would not be expected to result in a reduction in youth smoking unless other components of the counter-marketing program were also influencing these youth. Different interpretations of short-term, intermediate, ad long-term outcomes occur.The logical sequence is the most more important outcome.

39. CDC�s Designing and Implementing an Effective Tobacco Counter-Marketing Campaign

40. Framework for Program Evaluation in Public Health