Implementing a Randomised Controlled Trial for the evaluation of Probation supervision

1. Implementing a Randomised Controlled Trial for the evaluation of Probation supervision Presenter: Dominic Pearson Collaborators: David Torgerson Cynthia McDougall Roger Bowles

2. Overview of Presentation • Citizenship programme • Commissioned evaluation • Local implementation constraints • Solutions adopted • Results of deployment • Conclusions • Methodological questions

3. Citizenship programme • Designed in County Durham Probation Area • Operational staff and managers, supported by a consultant (Professor Clive Hollin) • Structured one-to-one work • A planned response to assessed crime-related needs using specific modules (e.g. alcohol) • Aims to raise awareness, motivate to change and link in with external resources

4. Commissioned evaluation • Commissioner required a regional evaluation • 3 Areas, different profiles • Aims • Identify whether the programme reduces reconvictions • Identify whether the programme promotes engagement with relevant community provision • Identify the benefit-costs relationship

5. General barriers to an RCT in the National Probation Service • Ethical resistance to random allocation • Judiciary • Probation managers • Probation practitioners • Programme management concerns • Difficult for practitioners to manage cases in two different ways • Related quality control issues • Staff training: a randomised phased allocation of the programme = more training events

6. Specific local barriers to an RCT • Area A - had already implemented at the time of commissioning • Area B – had resource issues linked to workload climate • Area C – had performance management issues in some sectors

7. Area A (had already implemented) Retrospective design Area B (had resource/workload issues) Opted for a single Area-wide launch with senior management support (‘big bang’) – retrospective design Area C (had offices with performance issues) Opted for a ‘stepped wedge’ design with offices randomly allocated in sequence to begin programme deployment Office 1→ April 2007 Office 2 → June 2007 Office 3 → August 2007 Solutions adopted • Office 4→ October 2007 • Office 5 → December 2007 • Office 6 → February 2008

8. Area A 75% (3072 / 4078) Area B 27% (2499 / 9749) Office 4: 40% (14 / 35) Office 5: 31% (21 / 67) Office 6: 65% (11 / 17) Results of deployment- take up rates • Area C • 44% (188 / 426) • Office 1: 47% (26 / 65) • Office 2: 51% (50 / 98) • Office 3: 46% (66 / 144)

9. Conclusions on Implementation • Area A was a different case • Implemented in a previous period and over a longer time • Area B vs. Area C • Phased deployment produced better results at less cost • Area B needed to re-launch (‘big bang’ #2) • Progress in Area C could be monitored and managed • Differential take up can be adjusted for in the analyses • Lesson in the importance of discussing evaluation at the time of implementation

10. Methodological questions • Overall / Regional evaluation • 3 stages • Different methodologies and Area profiles • Citizenship programme is the constant • Positive results from the first stage evaluation • Results from the (deficient) retrospective designs will be illuminated by the RCT • How will the effects associated with the different methodologies compare?

11. Thank you for listening • We welcome your comments and thought-provoking questions!