ESRC First Year Student Conference 2015

1. ESRC First Year Student Conference 2015 Carrot flavoured sticks: embedding evidence in government policy Richard Thurston Deputy Chief Social Research Officer Welsh Government

2. 1. The role of evidence (in theory) Governments need robust evidence to... • Anticipate issues/trends that governments will need to respond to • Monitor social and economic change and policy implementation • Appraise which policy/delivery options are likely to be most cost effective • Tell us what key stakeholders think and understand • Know whether or not policies are effective and having an impact (and if not, why not) • Help to identify and apply lessons to improve policy over time (design and implementation) • Ultimately – allowing more informed decisions

3. 1. The role of evidence (in theory) The origins of evidence-based policy-making • Not a new idea, but a relatively new ‘paradigm’ in government • New Labour – out with ideological government, in with rational decision-making • Prominent not only in government • Embodied in Civil Service Code • As ubiquitous as ever?

4. 2. The role of evidence (in practice) EBPM has some conceptual difficulties • Evidence-based, evidence-informed, evidence-aware – what’s the difference? • No linear model of policy development – not formulaic • Evidence mainly about the past, policy mainly about the future • Detractors argue there must be policy before there is evidence (value judgements play a big part)

5. 2. The role of evidence (in practice) All of the below constitute evidence-based

6. For me, politics shouldn’t be some mind-bending exercise. It’s about what you feel in your gut There is nothing a politician likes so little as to be well-informed. It makes decision making so complex and difficult.

7. 2. The role of evidence (in practice) Views of politicians vary…

8. 2. The role of evidence (in practice) Views of politicians vary… ‘Social science research evidence is central to development and evaluation of policy ….we need to be able to rely on social sciences and social scientists to tell us what works and why and what types of policy initiatives are likely to be most effective’

9. 2. The role of evidence (in practice) Views of politicians vary… "...we, as politicians, are elected by people to make a difference, to make things happen; we look at the evidence, but there comes a time when you say, 'you use your judgment‘.  There could be other things where there is evidence that something works [but] for perfectly good policy reasons, we say it is not the thing we want to do”

10. 2. The role of evidence (in practice) Views of politicians vary…….. “For me, real value is about being absolutely focused on the outcomes we are achieving out there in the real world…Tell us the facts and set out the evidence that underpins the options” First Minister (presentation to senior civil servants, 2011)

11. 2. The role of evidence (in practice) Source: Davies (2004:7)

12. 2. The role of evidence (in practice) The ROAMEF cycle: a role for evidence at every stage?

13. 2. The role of evidence (in practice) The ROAMEF cycle: a role for evidence at every stage? Rationale – e.g. evidence that there is a problem/question to be addressed, evidence that government intervention has the potential to help; Objectives – e.g. evidence on what aspects of the problem can be remedied, over what timescales and in what quantities; Appraisal – e.g. comparing different options to deliver the objectives, in terms of how likely they are to work and how much they cost; Monitoring – e.g. gathering monitoring data, and other evidence, during implementation to feed in to evaluations; Evaluation – e.g. an impact evaluation to determine whether the intended effects of a programme have been realised, or a process evaluation to determine how that happened; Feedback – e.g. using evaluation findings to determine the future of a programme or inform the design of a new one.

14. Fit with the Government’s programme and Ministerial priorities Impact on the people of Wales and the supporting evidence Cost of our investment Mechanisms available to incentivise change Management of the work 2. The role of evidence (in practice) The Welsh Government applies five principles that should be considered at every stage of policy making:

15. 2. The role of evidence (in practice) Analytical professions in Government What is KAS? • GSR, GES, GSS, GIS, Operational Research, Library, Chief Scientific Officers • Collection, analysis and presentation of data and research for policy makers and the public. • Professional standards (quality standards, ethics, publication, etc) • Analytical teams for each dept. • Cross-cutting work (e.g. National Survey, Corporate Research, Treasury analysis)

16. 2. The role of evidence (in practice) The sorts of things we do (in GSR) What is KAS? • Describe situations factually and how they change over time • Monitorspecific aspects of policies/programmes and how they change over time • Appraise proposed policies/options to establish their likely efficacy. • Evaluate how things worked in practice to learn lessons and assess impact • Calculate value for money through assessing cost effectiveness, efficiency and effectiveness • Estimate social impacts of policies/programmes • Understand and measure behaviour change

17. A departmental view … • Education & Skills Analytical Team • DfES Evidence Plan • DfES Evidence Board • Minister for Education and Skills • Ministerial Policy Board • Delivery Board • Ministerial Advisory Group

18. Education Examples – data collection and analysis • Statistics • Economic appraisal • Surveys • Secondary data analysis • Data linking • Qualitative Research • Research syntheses • Policy evaluations • Programme trials • Explorative Research

19. 3. The challenges – what makes the job interesting? The trade-off between quality vs timeliness • Good science can take a long time • But governments need to make quick decisions • Challenge in reconciling these conflicts – how? • Expediting - rapid evidence assessments, interim evaluations, accessing expertise, shifting resources • Giving best quality advice possible, with limitations clearly laid out

20. Counterfactual (no intervention) Change in impact measure New initiative 3. The challenges – what makes the job interesting? The limits of causality and certainty • Understanding the impacts of policy is difficult Post-new initiative trend Impact?

21. 3. The challenges – what makes the job interesting? The limits of causality and certainty • Understanding the impacts of policy is difficult • But policy makers need to understand in order to make good decisions and spend money wisely • Difficulties exacerbated by the timeliness vs quality trade-off and methods of policy implementation • We endeavour to: • carry out sound impact evaluations • influence the way policies are designed and rolled out • advise against ineffective policy options

22. 3. The challenges – what makes the job interesting? Cultural challenges around evidence and research • Awareness of the importance of evidence varies (e.g. “We don’t need to evaluate that, we know it works”). • Objectively assessing policy does not always fit with political priorities (e.g. “We’re keen to demonstrate the success of this policy”). • Balanced view of effectiveness sometimes seen as a negative (e.g. “We’re reluctant to do that in case it shows the policy isn’t working as well as we thought”) • Sometimes the ‘wrong’ evidence can be the most powerful (e.g. “Statistics show we’re doing a lot more of that, so we’re definitely having an impact”).

23. 3. The challenges – what makes the job interesting? Resources, resources, resources • Good quality social science takes time and financial commitment (often lots!) • Balance between commissioning and in-house work to keep skills sharp • Budgets are increasingly under pressure across the board – more research funding = hard sell • We continue to find creative ways to boost resources (like PhD Internships)

24. 3. The challenges – what makes the job interesting? The counter-arguments • Financial pressure means the question of ‘what works?’ is more important than ever • Hard to answer this question without rigorous social science • Even the most costly research is less costly than ineffective policy • More sophisticated approaches yield better estimates of impact and cost – pursue the most rigorous approach possible especially when equipoise exists

25. Enablers? Email: Richard.thurston@wales.gsi.gov.uk Web: http://wales.gov.uk/statistics-and-research/?lang=en