Stopping Trials for Futility

1. Stopping Trials for Futility RSS/NIHR HTA/MRC 1 day workshop 11 Nov 2008

2. Programme 11.00 - 11.05 Welcome 11.05 - 11.20 Introduction 11.20 - 11.40 Andy Grieve 11.40 - 12.00 Carl-Fredrik Burman 12.00 - 12.20 John Whitehead 12.20 - 12.50 Questions 12.50 - 1.30 Lunch 1.30 - 2.45 Workshop discussions 2.45 - 3.15 Feedback and discussions 3.15 - 3.30 Summary and final voting

3. Workshops • There are 4 workshop groups • 2 in the main lecture theatre • 1 in the Nightingale room • 1 in the Council chamber • All should have been allocated to a workshop group and location • These discussions will be facilitated by • Jenny Freeman (University of Sheffield) • Jon Nicholl (University of Sheffield) • Louise Dent (HTA) • Andrew Cook (HTA)

4. Workshops • Specific questions to be addressed are: • Should we plan to do futility analyses in publicly funded trials? • In what circumstances are they appropriate? • Should funding bodies insist upon them before granting trial extensions? • Three case studies have been supplied to facilitate discussions

5. Futility analysis Jon Nicholl ScHARR, University of Sheffield

6. Publicly funded trials Publicly funded trials are supported by MRC/NIHR Early phase trials are supported by MRC, later phases by NIHR Phase I – IIa DPFS (MRC), Phase IIb – EME (MRC/NIHR), phase III/IV – HTA Trials are also supported by NIHR through RfPB, applied programme grants, etc

7. HTA • HTA budget = £88m pa at maturity • Nearly all to be spent on trials • Typical trial design • Parallel group RCT of cost-effectiveness • 2 arm head-to-head or 3 arm with placebo • Multi-centre • 3-5 years (ts=0.5yr, tr=2yr, tf=1yr, ta=0.5yr) • £1m - £2.5m • N= 400 - 4000

8. Futility designs Why is this? Should futility analyses be designed into HTA trials? If so, in which trials? And when and how should they be done? Futility analyses are not uncommon in Pharma trials But no HTA funded trials have planned a futility analysis

9. What is a futility analysis? • A futility analysis is a calculation made during the course of a trial of the probability that the trial will produce helpful results • A trial is judged futile if the probability is too small • Helpful results are usually taken to mean ‘statistically significant’ results for the primary endpoint • So the calculation usually takes the form of a power calculation conditional on the results observed.

10. Should HTA do futility analyses? • YES • Opportunity cost - spending public money on futile trials is wrong because the money could be spent on other trials • It is ethically wrong to continue to recruit patients to trials with little hope of achieving helpful results • NO • All well conducted trials provide valuable evidence • The costs of designing trials with a planned futility analysis would outweigh any savings

11. Futility analyses in HTA trials Assuming they are not ruled out in principle • In what types of trial • In what circumstances • When and how should we do them?

12. 1. Types of trial So are futility analyses inappropriate in head-to-head trials? • In placebo controlled trials, futility suggests no worthwhile benefit (ie little or no chance of finding evidence that the intervention is better than placebo) • ie the treatment is futile as well as the trial • In head-to-head trials this isn’t true • The treatment may be far from futile

13. 2. Extensions 54% of MRC/HTA trials seek extensions Usually due to long set up time and/or slow recruitment 55% of trials with extensions still don’t achieve N Cost of extensions = £50k - £1m

14. Extensions Is it appropriate to ask for a futility analysis in extension requests? Is this the right form for the futility calculation? What level of conditional power should be taken to mean its futile to continue? • HTA currently asks for a ‘futility’ calculation “If the aim of the extension is to detect a difference between the interventions, please state the conditional power (two-sided 5% level) given the data so far, of detecting a currently plausible treatment effect”. • Almost never done as intended or refused

15. 3. When and how • Futility analyses should be conducted in large expensive trials in which tr >> tf and in which early stopping could save substantial resources. • A futility analysis should be conducted at the time that the expected resource savings are maximised • Expected resource savings = (resources not spent – shut down costs) x (prob shut down) • If a conditional power calculation is used, use a conservative rule for futility, eg conditional power < 20% • Take other information into account before deciding on stopping or not extending

16. 4. Unconditional power calculations Achieved sample sizes in 122 MRC/HTA trials (from Campbell et al, HTA 2007, 11 (48))

17. Problem is this removes any incentive to maximise recruitment and increases N, T, and £C • We could design the trial with fixed T (and £C) and target sample size N, but variable achieved size n • So instead of choosing N so that Prob (Z ≥ U | H₁, N) = 1 - β • N is chosen so that ∑ Prob (Z ≥ U | H₁, n).Prob(n | N) = 1 - β • Extensions for sample size would then be redundant

18. Possible recommendations • Build in a futility analysis when: • N and £C are large • There is a placebo arm • Tr >> Tf • Carry it out when the expected resources saved are maximised • Use a conservative criterion for deciding futility, eg conditional probability <20% • Always carry out futility analysis in extension requests unless potential savings < £?

19. Ask the audience

20. a.m. results:Should we plan to do futility analyses in publicly-funded trials? • Yes – for all appropriate trials (built into design) • Yes – but only for extension requests • Never • Undecided

21. a.m. results:In what circumstances are they appropriate? • Only in appropriate placebo-controlled trials • In both placebo-controlled and head-to-head trials • Undecided

22. a.m.results: Should funding bodies insist upon futility analyses before granting trial extensions? • Yes • No • Undecided

23. a.m. results:What is futile?How low can the conditional power be before stopping a trial? • 1% • 5% • 10% • 15% • 20% • 30% • 40% • 50% • It depends

24. p.m. resultsShould we plan to do futility analyses in publicly-funded trials? • Yes – for all appropriate trials (built into design) • Yes – but only for extension requests • Never • Undecided The percentage in the first category increased to about 72% (from 53%), as a result of the undecided making up their minds

25. p.m. results:In what circumstances are they appropriate? • Only in appropriate placebo-controlled trials • In both placebo-controlled and head-to-head trials • Undecided There was a slight shift in the undecided category into the first category

26. p.m.results:Should funding bodies insist upon futility analyses before granting trial extensions? • Yes • No • Undecided There was a shift from the undecided into the no category so that the numbers answering ‘yes’ and ‘no’ were both in the low 40s

27. p.m.results:What is futile?How low can the conditional power be before stopping a trial? • 1% • 5% • 10% • 15% • 20% • 30% • 40% • 50% • It depends The ‘It depends’ category increased from 72% to 84%

28. Thanks • Jon Nicholl, for suggesting the idea • The speakers: Andy Grieve, Carl-Fredrik Burman and John Whitehead for agreeing to speak • Paul Gentry and the RSS for organising things here and providing the venue • Louise Dent and Andrew Cook for facilitating the workshops • MRC and HTA for funding the event