What Works? Targeting, Testing & Tracking (Triple-T) AMIMB, Glazier’s Hall 29 October 2013

1. What Works?Targeting, Testing & Tracking(Triple-T) AMIMB, Glazier’s Hall29 October 2013 Lawrence W. Sherman Institute of Criminology Cambridge University

2. Forecasting Storms • 1987 –Big False Negative • 2013—Big True Positive • Week in advance • Protective actions taken • Huge reduction in harm • Same in Eastern India 2013 • Who was the hero?

3. Who Was Vice-Admiral Robert Fitzroy?

4. 1819: Weather Was All Words • Fitzroy changed the business model • ADDED numbers into it • Sharpened the words with scales, % • Improved accuracy over “experience” • Transformed qualitative forecasts • Into quantitatively-supported statements • Now we call them “algorithms” • See things coming much sooner

5. Meteorological Office of the Board of Trade 1854 • Job: Publish Weather Data • Not Predict It • Fitzroy invented term “forecasting” • Controversial idea • Not 100% accurate • Royal Society Attacked It • Budget Cut; forecasts stopped • Fitzroy committed suicide

6. Prevention Finally Won • Falling Barometer  Gale Warnings • Fishing ships told to stay in port • Owners objected; Practice discontinued • Deaths from gales; Prevention revived

7. What’s YOUR Job? • Measuring Harm? • Preventing Harm? • Reducing Harm? • Correcting Harm? • Redressing Wrongs? What is your business model? Nor for making money. Just for doing the job.

8. Words--not many Numbers • Every prison is different • Every human life is important • All rights must be protected BUT.... • Which prisons are worst? • Which are about to explode in violence? • Which will have the most suicides? • How can algorithms help prevent harm?

9. AMIMB Website Einstein Quote • Using same method • Expect different result? • Is there a new method IMBs could use? • Are there other business models? • Could they get better results? • Could they ADD numbers to add VALUE?

10. Evidence-Based Practice 1. An Idea: “...Practices should be based on scientific evidence about what works best.” (Sherman, 1998) 2. An Analytic Framework “A standard for all....strategies: using scientific evidence to target, test and track ...[all] practices” (Sherman, 2013)

11. Words AND Numbers • AMIMB Practical Guide Has Many Words • Many numbers requested • Not clear how they are to be gathered • Whether comparisons could be reliable • What is the logic model by which • EACH IMB REPORT CAN REDUCE HARM?

12. Same Problem for HM Inspectors • Police since 1856 • Prisons? Probation? • “Efficiency and Effectiveness” • Case by case • Word by word • HMI Police developed a “report card” • But measures were not scientific • Not “risk-adjusted”

13. Daniel Kahneman

14. QUICK! What emotion does this person feel?

15. QUICK! How Much is This Product? 17 X 24 = ???

16. Thinking, Fast and….Slower • Reading the face—fast! • Multiplying two numbers—slow (408) • Why is that important to monitoring? • It adds theory about how we think, decide, act

17. System I FAST Intuitive Automatic Effortless Associative Rapid Opaque Process Skilled System II SLOW Reflective Controlled Effortful Deductive Slow Self-Aware Rule-Following Two Cognitive Systems:Overlapping

18. Good News, Bad News Good News: • Most decisions are made with System I • System I conserves energy, time • Most System I decisions are right--driving Bad News: • Many important decisions are “wrong” • Many could be “right” if we used System II • We resist System II because it is “costly,” tiring

19. Slow Thinking About Strategy Targeting Aiming for biggest impact Tracking Measuring BOTH policing and crime Testing Deciding what works

20. Targeting---PREDICT

21. Began Career with Paul Meehl (1954) 200 Replications Since Then Formulas always beat (60%) or equal (40%) human judgment—latter is far more costly Crime, parole, air pilot errors, credit risk Wine value forecasts Cot Death risk Paul Meehl Clinical vs. Statistical: Kahneman Chapter 21

22. 3 ways of Predicting Qualitative 1. Clinical (System 1~2?; not transparent) Statistical: 2. checklist with validation 3. supercomputer data mining

23. High Risk (2%) Neither High nor Low Risk (38%) Low Risk (60%)

24. High Risk 2% vs. Bottom 60% Two Years From Forecast Date Charges for Any Offence 8 X more Charges Serious Offence 10 X more Charges Murder or Attempt 75 X more

25. Testing • Comparing two methods • Same kind of problems THEN,.... ASKING: • Which one works better? • Which one costs less? • Which one gets best result for same cost?

26. Testing Defined • A Fair Comparison • Between two different methods • E = Experimental Method (new) • C= Control Method (current) • All else equal • Which one is better? • By what criteria?

27. Not This Kind But This Kind What Kind of Evidence?

28. But what is good evidence? • What constitutes a good test? • Many bad tests • More intuitive than quantitative • “Illusion of Validity”

29. Restorative Justice Experiment:Did Program Cause Crime Drop?

30. Post Hoc Ergo Propter Hoc? After this, then because of this? Inferring Cause From Trend?

31. Or would it have dropped anyway? • “Natural” Trend • “History” • Other factors • “Spurious” explanations • That can be eliminated—give confidence All ruled out by randomized controlled trials (RCTs)

32. Randomized Controlled Trial RCT:COMPARISON or NET difference

33. CONTROL Group • A sample that measures what would happen at the same time and place without the intervention being introduced to an otherwise identical EXPERIMENTAL Group

34. AFP/ACT ANU Experiment:Did Program Cause Crime Drop?

35. Post Hoc Ergo Propter Hoc? After this, then because of this? Inferring Cause From Trend?

36. Or would it have dropped anyway? • “Natural” Trend • “History” • Other factors • “Spurious” explanations • That can be eliminated—give confidence All ruled out by randomized controlled trials (RCTs)

37. Randomized Controlled Trial RCT:COMPARISON or NET difference

38. CONTROL Group • A sample that measures what would happen at the same time and place without the intervention being introduced to an otherwise identical EXPERIMENTAL Group

39. Tracking

40. Tracking in Politics • Obama’s “Cave” • Tracking volunteer activity—500 offices • Contacting voters • Determining who supports Obama • Determining when & how they will vote • Arranging transport to the polls • Scheduling election day drivers • “Ground Game” cost $100 Million

41. 3. Tracking in Prisons • Liebling’s Quality of Prison Life Measures • What are the trends in --inputs (resources) --outputs (activities) --outcomes (results)

42. AMIMB-NOMS COMPSTAT?

43. Tracking With Evidence • Discuss • Criticize • Problem-Solve • Use and misuse of data • Refinement through trial and error • Technology making change easier

44. Sherman: 1998

46. The Rise of Evidence? • More Evidence is Available • More Evidence is Demanded • Budget cutbacks make evidence relevant • But there are only early adopters • A tipping point may some soon • The pace is quickening

47. How Can IMBs Be Evidence-Based? • Quantify current reports • Design new—fewer—measures • Test different models of IMB work

48. What Works?Targeting, Testing & Tracking(Triple-T) THANK YOU Lawrence W. Sherman Institute of Criminology Cambridge University