‘Trying it out’ Review of effectiveness of government pilots

1. ‘Trying it out’ Review of effectiveness of government pilots Roger Jowell Centre for Comparative Social Surveys City University, London with Annette King, Government Chief Social Researcher’s Office, Cabinet Office 1 July 2004

2. Roger Jowell (Chair), City UWaqar Ahmad, DETRSue Duncan, PM’s Strategy UnitJohn Fox, DoHEdward Page, LSEMichael Richardson, DWPJudy Sebba, DfESAnn Taggart, HM TreasuryRobert Walker, Nottingham UPaul Wiles, Home Office CABINET OFFICE TEAM: Phil DaviesAnnette KingRebecca StanleyTess RidgeLucy Woodward Panel of Enquiry

3. Components of the Review Expert Workshop Literature Review Postal survey of 11 Departments Interviews with senior civil servants Interviews with selected ministers Case studies

4. Advantages of policy pilots Evidence-informed policy and delivery Prior assessment of risks and benefits Identifying fault-lines and potentially expensive failures Promoting innovation

5. Forms of evaluation Summative and formative Systematic Reviews Before-and-after studies Quasi-experimental designs Randomised controlled trials – of individuals or areas

6. Obstacles Issues of ‘clinical equipoise’ Election cycles and legislative time frames Lack of resources Hazards of false positives/negatives

7. Consequences of present practice Pilots that aren’t pilots Insufficient planning and training Overlooking what is already known Inadequate rigour

8. Conclusions 27 recommendations, adding up to case for … • greater use of ‘genuine’ pilots across government • early discussion of ethical and practical difficulties • pilots as part of continuous policy evaluation • pilots must run their course • distinction between negative results & failures • more rigour, more RCTs, more independence, more time, more resources • “an indispensable tool of modern government”