Assessing the Contribution of Policies to Outcomes – Logic Modelling and Contribution Analysis

1. Assessing the Contribution of Policies to Outcomes – Logic Modelling and Contribution Analysis Jackie Horne, Alison Stout Strategic Research Strategy and Performance Division The Scottish Government Heather Doherty Research Manager Museums Galleries Scotland

2. Learning Outcomes • Distinguish between inputs, outputs and outcomes • Understand the theory and practice of logic modelling and contribution analysis • Practice developing a results chain and a logic model to help assess how and whether policies are delivering on outcomes • Practice developing indicators to measure progress against outcomes

4. Outcomes approach Integral to: • Programme for Government • Spending Review 2011 • Government Economic Strategy • Renewing Scotland's Public Services - Priorities for reform in response to the Christie Commission

5. The outcomes approach It is widespread…… • Countries that have implemented an outcomes based approach include Canada, New Zealand, Unites Stated (in various forms across different states), England, Singapore and Malaysia, almost all other OECD member countries, many developing countries and a number of international agencies and NGOs. But the language differs……… • Outcome governance, outcomes management, accountability for outcomes, results based management, but, overall, the reform agendas are similar.

6. The outcomes approach There are commonalities: • they generally involve greater devolution of power and decision-making and commonly adopt some form of performance management and reporting system (e.g. Virginia Performs, New Zealand governments’ Statements of Intent, Canadian Government’s whole of government framework). It takes time is a common message…. • Evidence suggests at least 4-5 years of consistent effort is required to embed an outcomes approach —and many organisations have been at it much longer.

7. ExerciseDistinguishing between inputs, activities,outputs and outcomes

8. Understanding the relationship between policies and outcomes – single intervention

9. A simple results chain Increase in healthy life expectancy Lower blood pressure, improved mobilityamong over 65s Increased levels of physical activity in over 65s Over 65s, especially those not doing enough exercise More accessible exercise classes and facilities for over 65s. Invest in exercise facilities and classes for over 65s Staff, equipment, leisure centres, etc.

11. ExerciseResults chain

12. Multiple policies - logic models and contribution analysis • Move beyond results chains to consider inter-linkages between policies and outcomes • Consider strengths (and weaknesses) of the contribution of different policies to outcomes • Assess and measure progress towards purpose targets and national outcomes • Inform cost-effectiveness and resource allocation – decisions on priorities • Highlight strengths and weaknesses in evidence base

13. Steps of contribution analysis Identify outcome Logic modelling Evidence gathering Performance story

14. Uses of logic modelling and contribution analysis • Evidence effectiveness of a defined policy area (Population Purpose Target) • Work out public sector partner organisations’ contribution to policy area and high level outcomes (Alcohol – Health Scotland) • Identify cross office and agency contributions to National Outcomes (National Outcome 12) • Develop indicators (CYP) • Business planning and indicators (Justice)

16. High level outcomes Intermediate outcomes Increased priceReduced incentives Increased sobriety & stability Short-term outcomes Compliance with laws Increased detection rate Alcohol industry Adults with alcohol problems Reach Licensed trade Drivers Taxation,displays, promotions, advertising Enforcement actions Outputs Random breath testing Addiction services Enforcement of planning controls & licensing laws Industry regulation Detox, Intensive support Activities Enforcement of drink driving laws SG, UK govts, EU Inputs Local authorities Vol orgs Police Alcohol – Partner Contributions to Outcomes Improved mental wellbeingReduced inequalities in healthy life expectancyReduced inequalities in alcohol-related deaths and hospital admissions BehaviourReduced alcohol consumption levelsLess drunkenness; less drink-driving EnvironmentsPhysical:Reduced exposure to alcohol-related hazardsEconomic: Reduced availability/affordability of alcohol Social: Drunkenness less attractive; sensible drinking thenorm Service uptake & engagement Understanding risks, attitudes to drinking Hazardous and harmful drinkers General public - targeted Preventive services Sensible drinking messages Media campaigns Brief advice Scottish Govt NHS

17. 5 years 10 years 15 years 20 years 25 years + Improved understanding Actions planned at ecosystem level Greater consensus Planning is less adversarial integrated approach (amongst Government, on which climate Fully to surface public bodies and landowners) of: change impacts to tackle water management National planning policy context helps to - impacts of Climate Change and how to address More co-ordinated approach to flood ensure that development plans and [deleted CC adaptation bullet as this is them risk management management decisions protect and the same] enhance natural and built environments - relative flood risk across Scotland and create opportunities for their - the role of land use in managing flooding and enjoyment river basin quality - the marine environment and ecosystems - how to achieve 'good environmental status' in Processes land and marine environments Better partnership working with and between stakeholders and key agencies Plans and strategies in place for: - embedding CSGN into development plans under NPF2 - flood risk management (also co-ordinated with river basin planning) - surface water management - driving decisions on use of marine environment Greater public understanding and awareness Greater appreciation Widespread public understanding value The whole of central Scotland is seen as of the role and value of: People and communities direct place of choice to live and work of: of the environment (built and natural, - the impacts of climate change and indirect benefits and services The green network is valued for its - greenspace and protected areas and including marine) for ecosystem health and - how soils store carbon (especially provided by biodiversity and healthy contribution to enabling life styles which opportunities they provide our economy amongst land managers) functioning ecosystems Public - responsibilities towards the built and natural - flood risk (and people take responsibility rely on less resource use Awareness Public take responsibilty for protecting environment for their own protection) themselves against flooding - flood risk and of actions people can take Knowledge informs land use decisions (at community level) Tourism and recreational use increases Participation in outdoor recreation equals People and communities are strongly Recreation, Higher levels of satisfaction reported with tourist or exceeds national average with greater enjoying and protecting involved in Enjoyment visits rate of increase in deprived areas biodiversity and healthy ecosystems Health and well-being improves Health and well-being continues to improve and Well-Being Land: Land: Land: Land: Land: - greening of derelict land increased - green networks integrated with buildings, - new infrastructure developed to - loss of species and habitats halted - rate of woodland development increased paths and transport infrastructure withstand effects of climate change - ecosystems in protected natural areas - access widened for all groups - increase in quality of greenspace - land based businesses are successful, offer optimum levels of services - 72% of water bodies at Good Ecological Status - losses in biodiversity minimised using natural resources sustainably - land is better used, and use is better - water abstracted in a more efficient way - 95% of features in special conservation - 97% of water bodies are at Good integrated areas in favourable condition Ecological Status - planning developments more - protected areas are part of healthy sustainable Marine: - 77% of water bodies are at Good Protection and Ecological Status ecosystems - Scottish Sustainable Communities are - protected areas are designated Enhancement - land management improved, leading to - populations of rare and endangered exemplary models for 21st century - areas are managed to improve health and protection of river basins species are maintained living diversity of environment and species - places actively managed to be part of - reduced number of areas assessed as - Scotland's communities are resilient to - businesses are economically and ecosystems being at significant flood risk change environmentally sustainable - Scotland taking advantage of positive Marine: opportunities from climate change - species are healthy and diverse NO 12 - We value and enjoy our built and natural environment and protect and enhance it for future generations

19. National Outcomes- Justice 9 – We live our lives safe from crime, disorder and danger 11 – We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others 15 – Our public services are high quality, continually improving, efficient and responsive to local people’s needs

20. Unpacking theNational Outcomes – Justice

22. KEY TO RECOMMENDATION: RED – No further action Amber – Evidence unclear or lacking Green – Pursue further

23. Piloting contribution analysis in Museums Galleries Scotland

24. What did we pilot it on? • Hands On - An existing project that had come to an end in its current form

25. Where have we used it since? • Festival of Museums – A new programme that was about to start • Choices for Change toolkit – a decision-making toolkit that was developed collaboratively • Collaborating to Compete conference

26. What did we learn? • It was helpful to name our first attempt as a pilot

27. 2. It’s additional to any existing evaluation activity – rather than a replacement

28. 3. It was helpful to apply the tool retrospectively to familiarise ourselves with the logic model part of the proces

29. 4. BUT, more value in applying from the beginning of a project

30. 5. It gives an overview of a piece of work and its component parts in a way that we don’t get with any other method

31. 6. It can be applied to a small programme as well as a large one

32. 7. The logic model can be customised to suit our needs • To incorporate our evaluation framework • To incorporate other frameworks, for example Curriculum for Excellence • From project to project

33. 8. Using the logic model part of the process is valuable in itself, even if you don’t go on to further analyse each contribution

34. 9. It’s helpful to populate the logic model as a group (even a group of 2 people)

35. 10. Once you understand the logic model headings and what they mean in a contribution analysis context, populating the logic model becomes easier and quicker

36. 11. The logic model headings are very useful for revisiting the reason for your project or programme

37. 12. The type of work that fits well with this approach quickly becomes apparent

38. 13. Introducing contribution analysis to people works best with conversation and coaching, being given the documents alone to read is not enough

39. ExerciseLogic modelling

40. Indicators ‘’Imagine a car dashboard: an indicator is a warning light flashing on the dashboard. It is fed by one of many streams of data – maybe oil level, temperature, etc...It flashes when all is not well, suggesting we stop the car. The indicator “alerts us to something worthy of further investigation.” The Good Indicator Guide (NHS Institute for Innovation and Improvement)

41. Early Years Outcomes and Indicators Framework • Core set of indicators to monitor progress at a national and local level www.scotland.gov.uk/Topics/People/Young-People/Early-Years-and-Family/Early-Years-Framework/Implementation/Measuring-Practice • Developed using logic modelling • Criteria • easily available at sub national level (local authority or NHS Board) • collected regularly • strong evidence base • collected by a credible source • tell us more than the face value of the individual indicator • taken together, cover the whole pre-birth to 8 age range.

42. Early Years – Core Indicators • Teenage pregnancy rate (pregnancies among under 16 year olds (3 year average per 1000 relevant population)) • Appropriate birth weight for gestational age (low weight live births) • Breastfeeding at 6-8 weeks (% of newborn children exclusively breastfed at 6-8 weeks) • % parents who rate their neighbourhood as a good place to live (% of adults stating their neighbourhood is a 'very good' place to live) • Children in benefit dependent households (% of children living in households that are dependent on out of work benefits or Child Tax Credit) • % of obese children in P1 (estimated % of obese children in P1) • % of children with dental decay in P1 (% of children with dental decay in P1) • Children (pre birth to 8) referred to reporter on care and protection grounds • Number of children pre-birth to 8 looked after a) at home b) accommodated • % of young people in sustainable post school destinations (% of school leavers in positive and sustained destinations)

43. Exercise - Developing indicators How will the indicator be defined and measured? Where will the data come from? Will it measure absolute numbers or proportions? How frequently will the data be updated? Will you set an associated target, to drive forward change? What is a reasonable length of time to achieve the desired outcome? Is it clear what type of change in the data represents an improvement in performance? Are there any caveats/warnings/problems? e.g. missing data, perverse incentives

44. The outcomes approach: general findings Key organisational challenges: • ensuring visible and clear political and senior level leadership • unrealistic expectations about the timescale for change • ensuring co-ordination across policy areas and tiers of government • linking performance information and decision-making locally and nationally • most jurisdictions have struggled to link outcomes with budgets • accepting that outcomes are only one factor in decision making

45. The outcomes approach: general findings Key technical challenges • defining outcomes in a measureable way can be difficult • avoiding performance indicator overload • attribution of outcomes to government action per se, or to a given policy, programme or project is not straightforward • outcomes generally have a long-term focus, which can result in reporting time-lags • outcome measurement, data quality and reporting problems

46. Contact details • Jackie Horne (jackie.horne@scotland.gsi.gov.uk) • Alison Stout (alison.stout@scotland.gsi.gov.uk) • HeatherDoherty (heatherd@museumsgalleriesscotland.org.uk)