Pathfinder Kick-off Meeting Research Methodology Introduction, Cardiff 2 nd March 2011

1. Pathfinder Kick-off Meeting Research Methodology Introduction, Cardiff2nd March 2011 Richard Redfern, Steve Tarry 2nd March 2011

2. Welcome and Introductions from presenters and participants

3. Content: General introduction to the day Introduction to the research to be undertaken Evaluation overview Baseline and contexts for the research Project Officers’ role – the monitoring, recording, reviewing and reporting of data and relevant information

4. General Introduction to the Day: A recap on what was covered in day 1: Study participants, roles and responsibilities Pivotal role of Pathfinder Officers (POs) Aim of day 2: To present an overview of the planned research Explain how POs will help in developing the evaluation approach and populating the study Evaluation Framework Address any questions or concerns about what is proposed Begin work on developing some of the tools required How the day will run: Presentation of particular topics associated with the research methodology Discussion of PO role in particular and any issues arising Interaction required to ensure POs have confidence in what is required

5. Hand Out Agenda at This Stage

6. General Introduction to the Day (cont): Agenda: Before lunch: Introduction to the research and overview of evaluation approach Lunch 12.30 – 13.15 After lunch: Specific context for the study, the role of case studies and the work of the POs Finish at 16.15 Expected outcome: insight into what will constitute an appropriate evaluation programme and an clear understanding of PO role Required deliverables: Tools required to support POs in their particular role What will follow on: Day 3 - Exploration of individual sectors within the study Meetings with host authorities to being process of engagement

7. Introduction to Planned Research The objectives of the Pathfinder Community Research project are to: Generate robust evidence of which projects work in reducing greenhouse gas emissions; Support the trial of different approaches to community action on climate change; Evaluate the different ‘models’ of community based project interventions; Evaluate the delivery of the pathfinder project programme by the programme management team; and Assess the role of Pathfinder Officers in delivering programmes. These are to be achieved through the evaluation of 18 case study initiatives spread across 6 Local Authority areas. The exact nature and location of the Case Studies is to be determined during this scoping phase of the project this will define the structure and scope of the research activities undertaken in subsequent phases of the project

8. Introduction to Planned Research (cont) In each of the six Local Authority areas the following outputs are to be achieved by the end of the two year pilot: A ‘map’ of communities who are already engaged in action and those which have potential and/or have demonstrated interest in engaging in action in the future. A ‘map’ of the communitieswho could be persuaded and encouraged to take action A record of carbon reduction work being undertaken by the community Specific community groups identified as willing to reduce their carbon footprint Baseline data for the carbon footprint of the willing community A carbon reduction target of 10 to 20% in the willing community within two years

9. Research Approach The research will look at initiatives within different sectors Energy Transport Water Food Research will look at behaviour changes / effects achieved by initiatives both within and across different sectors ‘Theory of Change’ will be used to explore the differences noted and the logic behind what has transpired The detailed approach will need to reflect the specific nature of the initiatives that are taken

10. Participants to think of the nature of the initiatives that might be taken:EnergyTransportWaterFoodRecord these; thenReview with results of previous Brainstorm – as summarised on next slides

11. Possible Initiatives? Energy Energy efficient appliances Energy efficient equipment Energy efficient lighting Energy efficient homes Green buildings and homes Green business Green energy Retrofitting insulation Programmeable thermostats Energy conservation • Transport • Anti-idling • Biking • Carpooling • Fuel efficient vehicles • Mass transit • Telecommuting • Vehicle maintenance • Walking

12. Possible Initiatives? Water Drinking water Greywater Lawn watering Low flow showerheads Low flow toilets Maintaining wells Water tanks Water efficiency Water restrictions • Food • Local food • Fertiliser use • Reducing meat and dairy consumption • Biodiversity • Forestry • Pesticide use • Organic food • Reduce packaging

13. Theory of Change • Robust framework for considering: • What has changed ? • Why ? • In what context ? • Tailored to address complex interventions, packages or programmes • ‘Mapping’ produced to record and test underlying logic of investment: • Links between investment and short term outcomes • Hypotheses of links between short and long term behavioural change • Work with stakeholders to review mapping and recognise their role within individual ‘blocks’ and in relation to specific ‘links’

14. Context Input Output Outcomes Impact The frame-work within which an intervention is located What is invested, e.g. money, skills, people, activities What has been produced Short and medium term results Long-term outcomes Analysis of context Analysis of objectives Analysis of the logic of the intervention Logic Mapping – the Theory:

15. Logic Mapping – an Example

16. Hand out Spreadsheet example for capturing appropriate data from which to develop appropriate Logic Map

17. Practical Example - Specific to the Study Activities Short Term Outcomes Medium Term Outcomes Long Term Outcomes Increased cycle skills Increased cycle skills Reduced accidents involving child cyclists Reduced accidents involving child cyclists Reduced carbon emissions Reduced carbon emissions Increased propensity for parents to let children cycle Increased propensity for parents to let children cycle Cycle Training for Year 5/6 Children Cycle Training for Year 5/6 Children More positive attitude to cycling by parents/children More positive attitude to cycling by parents/children Increased risk awareness Increased risk awareness Increased cycling to school Increased cycling to school Poorly delivered training Course not well designed Low take-up rate Misses ‘at risk’ children

18. Role of Case Studies Case Studies will give us practical examples of Inputs and Outputs Inputs are the investment, resources and processes applied in the implementation of case study interventions, including Local Authority and other stakeholders’ time and investment. Accurately monitoring, analysing and reporting the full range of inputs will be a primary focus for the evaluation framework Outputs are the initiatives delivered, as a result of the inputs directed towards the Pathfinder programme. These will cover the range of intervention types (to be defined), including transport, energy, water and food

19. Categorisation of Initiatives Categorisation of initiatives: provides a focus for data collection assists when reviewing the transferability of findings about similar types of interventions. Typology of initiatives according to, for example: Primary objective Primary target population Scheme ownership Geographical scale .

20. The Research Programme Short term Engagement Identify possible Case Studies Identify data sources Set up evaluation processes Identify Indicators of success (and possibly failure) Medium term Data collection Look at inputs and outputs Review evaluation approach Longer term Interim Look at outcomes and impacts Review research approach Final Look at outcomes and impacts Address specific research questions

21. Q and A

22. Evaluation Overview - What is Evaluation? • “Evaluation is the post implementation assessment” of: • Programmes • Policies and strategies • Investment ‘packages’ • Individual schemes/interventions • Two key areas in evaluation: • Outcome or Impact evaluation • Process evaluation • Both apply to the Pathfinder Research

23. Rationale Feedback Objectives Evaluation Appraisal Monitoring Evaluation within the Programme / Project Lifecycle Implementation

24. Benefits of Evaluating

25. Relationship to Logic Mapping Impacts Realm of evaluation • Rationale – Policy context • Objectives – Local priorities for investment • Inputs – capital investment and human resources • Outputs – delivered interventions • Outcomes – changes in key indicators • Impacts – longer term changes Rationale Outcomes Traditional realm of monitoring Outputs Objectives Inputs

26. Phase 1: Evaluation Scoping This phase defines the specific focus of the evaluation and, importantly, the interventions to be included. It includes the development of the evaluation framework and its constituent parts. This phase will identify existing evidence gaps and design data collection activities to fill these. Initial stakeholder and community engagement will be undertaken and process evaluation work will be undertaken. Phase 2: Interim Evaluation Ongoing data collection and detailed process evaluation activities undertaken during and after the implementation phase of measures. This will include further engagement and action research activities. Phase 3: Outcome Evaluation Phases of Delivery

27. Phase One (Scoping) – Overview Evaluation Scoping Define interventions and packages Confirm evaluation objectives Select research questions Select core indicators Select evaluation indicators Select contextual indicators ToC mapping Stakeholder liaison Conduct ToC mapping Identify new data needed Define evaluation approach

28. Objectives of the Research The objectives of the Pathfinder Community Research project can be grouped around three general themes: Outcomes: Generate robust evidence of which projects work in reducing greenhouse gas emissions; Engagement: Support the trial of different approaches to community action on climate change; Evaluate the different ‘models’ of community based project interventions; Processes: Evaluate the delivery of the pathfinder project programme by the programme management team; Assess the role of Pathfinder Officers in delivering programmes. The nature of research questions reflect the specific theme under consideration

29. Participants to think about the nature of questions / hypotheses under each objective on preceding slideRecord; thenCompare with what is on the following slides

30. Research Questions Objective 1: To generate robust evidence of which projects work in reducing greenhouse gas emissions Questions in terms of impacts: What levels of greenhouse gas emission reductions have been achieved? What are the key techniques/factors/activities that have reduced green house gas emissions? How do the reductions vary between investment themes? What are the timescales for outturn reductions?

31. Research Questions Objective 1: Questions in terms of behavioural change contributing to outcomes: What changes in behaviour have been generated, how and why? Are these sustainable? What are the key drivers and motivators for behavioural change? Do they vary in the different themes? Do these vary between locations and population sub-groups? Which approaches and techniques have generated the highest levels of behavioural change? To what extent has investment generated low-carbon communities? How was the long term sustainability of the project planned for and secured?

32. Research Questions Objective 2: To support the trial of different approaches to community action on climate change Questions in terms of community engagement: What approaches have been adopted to engender community action? Which approaches achieve greatest community engagement and support and why? Which project theme generated the greatest level of community engagement and why?

33. Research Questions Objective 2: Questions in terms of penetration and sustainability: How does the delivery of impacts of different approaches vary between locations, across socio-economic groups and why? What are the key factors in a given approach that accelerated success? What are the key factors that resulted in less than expected good results?

34. Research Questions Objective 3: To evaluate the different ‘models’ of community based project interventions What are the different models/approaches to community based project development? Which models/approaches are the most efficient in terms of costs/benefits? What are the key processes involved in each model/approach and what lessons can be transferred to other locations/projects? Which elements of approaches are most effective/important in reducing greenhouse gas emissions and why? Which approaches generate the best advancement in community knowledge and skills in addressing climate change? How do these vary between locations and case studies?

35. Research Questions Objective 4: To evaluate the delivery of the pathfinder project programme by the programme management team What co-ordination/leadership approaches by the programme management team facilitated dialogue and collaboration with key stakeholders and between Pathfinder Officers (POs)? What approaches acted as barriers to meaningful dialogue and collaboration? What support, training and direction provided for clarity of work by the POs? How effective was the Advisory Group in helping to deliver the work of the POs? What key strengths of the programme management team enabled action by the POs and what gaps/weaknesses acted as barriers? What other barriers challenged work of the whole programme?

36. Research Questions Objective 5: To assess the role of Pathfinder Officers in delivering programmes What role did the PO have in accelerating the work of the community projects they supported? What particular skills, knowledge and abilities of the PO supported the success of the community projects? What approaches to community participation taken by the individual POs supported community projects? What barriers did the POs face that acted as a barrier to the delivery of the work of the community project? How were they removed, if removable?

37. Phase One (Scoping) – Overview Evaluation Scoping Define interventions and packages Confirm evaluation objectives Select research questions Select core indicators Select evaluation indicators Select contextual indicators ToC mapping Stakeholder liaison Conduct ToC mapping Identify new data needed Define evaluation approach

38. Participants to consider what would make a Good Indicator Record, thenDiscuss individual merits

39. What Would Make a Good Indicator SMART: Specific Measurable Achievable Relevant Time based

40. What Would Make a Good Indicator (cont) But, what will be Practical?: What data are available? Who owns or has access to the data? What are the timescales for anticipated change?

41. Q and A

42. Lunch

43. Baseline and Contexts

44. Top Down Policy – Bottom up Community Actions Policy Current Baseline Emissions Future Target Actions

45. What Differences are Made as a Result of Pathfinder Officer Actions? Project might progress at a faster pace – with impact felt sooner How to assess what would have happened without engagement? Informed by ‘backward’ review and comparison with other similar projects without PO engagement

46. What Differences are Made as a Result of Pathfinder Officer Actions? Overall impact of Project might increase The availability of data for reviewing the baseline / activities prior to PO engagement will be a factor in the selection of case Studies

47. Q and A

48. Phase One (Scoping) – Overview Evaluation Scoping Define interventions and packages Confirm evaluation objectives Select research questions Select core indicators Select evaluation indicators Select contextual indicators ToC mapping Conduct ToC mapping Stakeholder liaison Identify new data needed Define evaluation approach

49. Monitoring and Recording Following initial engagement with Stakeholders Focus will be on collating basic ‘headline’ data for each potential case study (within a long list of, say, 120) This will enable actual case studies to be selected for evaluation Criteria to be determined to facilitate the selection of final case studies Criteria should allow comparisons to be made across candidate case studies Should aim for a representative cross section of case studies, covering all relevant sectors and differing approaches to community involvement, project development and delivery More detailed data will then to be recorded for selected case studies, to fully define and characterise the case study in advance of the evaluation phase The actual basic data to be collated will reflect the nature of the initiative and the criteria used for choosing between candidate case studies

50. Participants to brainstorm some of the criteria / data needsRecordHandout initial draft schema for reviewing projects (seeking financial support under sustainable communities programme)Participants to take this away and provide feedback on how this can be adapted to the needs of the Pathfinder research