1 / 19

Session C: Evaluation

Session C: Evaluation. James Farnham 3 June 2009. Content. Context for evaluation Principles of evaluation Stepped approach to evaluation. The context for evaluation. Evaluation benefits. Continuous learning and improvement improved effectiveness of marketing programmes

rio
Download Presentation

Session C: Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Session C: Evaluation James Farnham 3 June 2009

  2. Content • Context for evaluation • Principles of evaluation • Stepped approach to evaluation

  3. The context for evaluation

  4. Evaluation benefits • Continuous learning and improvement • improved effectiveness of marketing programmes • improved marketing efficiency • Better knowledge and insight for how to influence consumer behaviour • Accountability • More informed budgeting process • Better forecasting of outcomes • better management of expectations re results • input to ‘policy levers’ • enhanced credibility of social marketing

  5. Moving from a static, linear approach to a more dynamic, continuous improvement journey • Test-Learn-Refine (TLR) Evaluation Planning Evaluation Planning

  6. Evaluation perspectives Accountability Vs Test-Learn-Refine ‘What gets measured is what matters’ Scientific realists Vs Social constructionists ‘No single accepted best practice’ Theory Vs Practice

  7. A ‘line of sight’ to behaviour change is required across a range of measures Intermediate measures eg attitudes and response Behavioural change measures Reduced consumption Outtake measures eg understanding & knowledge Outcome/impact measures Reduced ARHA Output measures eg awareness Input measures eg toolkits, TVRs

  8. Principles of evaluation

  9. Ten marketing evaluation principles • Better evaluation will improve both our marketing results and our accountability – it is essential, not a ‘nice to have’ • Five themes • Culture • Process • Measurement • Influencing behaviour • Test-Learn-Refine

  10. Ten marketing evaluation principles Culture 1. Hard wire evaluation into every stage of the marketing process, focusing on active learning and rapid application 2. Evaluate every piece of activity at a proportionate level - between 5% and 10% of total spend

  11. Ten marketing evaluation principles Process • Develop a formal evaluation plan at the outset of the marketing process, including clear objectives. Each plan will address the following questions: • What will be evaluated? (i.e. what is the marketing activity about and in what Policy context does it exist?) • What aspects of the marketing activity will be considered when judging its performance? • What standards (i.e. type or level of performance) must be reached for the marketing activity to be considered successful? • What evidence will be used to indicate how the marketing activity has performed? • What relevant lessons from previous marketing activities can be applied? • What conclusions regarding marketing activity performance are justified by comparing the available evidence to the selected standards? • How will the lessons learned from the evaluation be used to improve public health effectiveness?

  12. Ten marketing evaluation principles Measurement 4. Collect robust, consistent data on marketing activity 5. Ensure all marketing evaluations focus on a range (c. 10) of KPI measures that are critical to success – both in terms of behavioural influence and ROI/cost-effectiveness. Avoid using awareness as an isolated measure of success Influencing behaviour 6. Put behaviour change at the heart of the evaluation thinking, capitalising on existing behavioural theory

  13. Ten marketing evaluation principles Test-learn-refine 7. Use a rigorous test-learn-refine process within and between channels 8. Try to test locally prior to full PCT exposure, establishing control areas if possible 9. Evaluation should be used to encourage innovation, not create risk aversity 10. Question activities that cannot be evaluated

  14. Stepped approach to evaluation

  15. Is it a step? NSMC Total Process Planning Model

  16. PHASE 1: SCOPE - gaining early clarity Step 2 Lessons learned from previous evaluations Step 4 Setting the evaluation budget Step 1 Form an Evaluation Team Step 5 Evaluation plan 1 Step 3 Clarifying objectives

  17. PHASE 2: DEVELOP – working up the overall approach Step 1 Using behavioural theory Step 2 Defining the behavioural goals Step 5 Briefing the implementation agencies Step 4 Writing the Evaluation Plan Step 3 Defining the KPIs

  18. PHASE 3: IMPLEMENT – iteratively designing the evaluation detail with all agencies Step 1 Evaluation development with implementation agencies Step 2 Designing and setting up the measurement approaches Step 3 Getting the most out of evaluation

  19. PHASE 4: LEARN – short term and long term Step 4 Sharing and capturing the learnings from the Evaluation Report Step 1 Immediate post-launch reporting Step 2 Interim Evaluation Step 3 Final Evaluation Report

More Related