Criteria for Assessing The Feasibility of RCTs

1. Criteria for Assessing The Feasibility of RCTs

2. Today’s Headlines: “Drugs education is not working” “ having reviewed research from across the world, a committee of doctors and scientists on the ACMD* concluded that the success of school-based schemes was "slight or non-existent" and could even be "counter-productive". *Advisory Council on the Misuse of Drugs (ACMD)

3. Why are we here? • We share the goal of improving well being for society and individuals • Interventions which we support as a society should benefit individuals and society and should not cause harm • We value evidence

4. Who am I? • Economist • Managing Director of Matrix Research & Consultancy Ltd • Member of the Campbell Crime and Justice Group • Member of the Cochrane Campbell Economics Methods Group • Advocate of improving the use of evidence to inform decisions

5. Who are you? • Politicians? • Policy advisers? • Civil Servants? • Practitioners? • Researchers? • Economists…..?

6. Important evaluation questions (adapted: Bain 1999) • Should it work?  Theory • Can it work?  Implementation • Does it work?  Impact • Is it worth it?  Value

7. Source of interventions • Historical practice (“we have always done it this way”) • Taught practice (“we teach people to do it this way”) • Innovative practice (“I try new ideas”) • Research (“I examine the theories”) the popular vote………………

8. Role of trials for new medical interventions • Idea • Basic science • Laboratory trials • Clinical trials • Licence • Approval (e.g. NICE) Average 10-15 years?

9. Particular challenges for trials in social science (adapted: Farrington 1983) • Pace of idea to practice • Theory base of treatment • Definition and scope of treatment • Context complexity • Community vs individual interventions • Hawthorne effects • Contamination • Outcome measures & measurement • Duration and decay

10. Arguments for Trials • Interventions can do harm • If undertaken well  trials minimize risk of bias • Basic hypothesis: “can we identify an effect with confidence?” • If results insignificant then: • There isn’t an effect; or • There is an effect but we haven’t detected it with confidence.

11. Arguments against trials • Epistemology: “the world doesn’t work like this” • Analytical: “there are too many analytical constraints” • Ethical: “you can’t deny treatment” • Legal: “you might be challenged if you deny treatments” • Logistical: “there are too many practical constraints

12. Research on Feasibility Research partners • Matrix Research & Consultancy Ltd • The Jerry Lee Centre, University of Pennsylvania. Research base • Collective research experience • Home Office funded study for OBPs

13. Analytical challenges: Hierarchy • Internal validity: can we attribute effect to intervention? • Statistical power: can we measure the effect with confidence? • External validity: can we generalise the results?

14. Hurdles to ensure Internal Validity • Participant selection targeting • Completion and attrition rates • Inconsistent treatments & treatment measurement • Multiple outcomes and inconsistent outcome measurement • Independent, contamination-free alternative, treatments or no treatment

15. Hurdles for Statistical Power • Understanding expected/required effect size? • And….Given this… • How to maximize statistical power given? • Heterogeneity of sample? • Completion rates? • Attrition rates? And…given this… • How many participants do you need?

16. Hurdles for External Validity Context / Characteristics: • Local service provider • Participants • Community Tension between correcting for these and the increased challenges of achieving internal validity and statistical power.

17. Ethical Hurdles • “denial of treatment is unethical” • “validity of informed consent”

18. Legal Hurdles • E.g. for offenders  tension between random assignment and requirements to sentence • Scope of legal challenge: • from those denied the intervention • where programmes exist and are perceived to reduce risk to public/ stakeholders

19. Logistical Hurdles • Quality of trial staff • Cost of a trial relative to the cost of the intervention and the value of the expected effect • Sample size required when completion rates are low and attrition post completion is high • Avoidance of Hawthorne effect

20. Are there Solutions? • Focus on particular well defined interventions • Appoint independent trained research teams • Randomise early • Separate control groups • Pick control treatments which enable effects to be measured • Manage and monitor implementation • Ensure programme stability • Determine sample size from desired effect size – look at the theory and related research • Measure outcomes consistently • Identify cost effective study design • Educate and inform participants and stakeholders

21. If these aren’t available? Only then ask yourself: “What is the next best alternative….and how can I minimize the risk of getting it wrong”

22. Final reflections • Researchers should recognise: • decisions need to be made in real time • all information can provide evidence • Users should recognise: • the risks of making the wrong decision increases as the quality of the evidence base declines • In social science we are addressing the needs of people and communities who are vulnerable