Benefit-Risk Analysis of Multi- Stage Adaptive Designs

1. Benefit-Risk Analysis of Multi-Stage Adaptive Designs Qing Liu, Ph.D. Kevin Liu, Ph.D. Johnson & Johnson PRD, L.L.C.

2. Outline • Motivation • Past work • Two-Stage Designs • Benefit Risk Evaluation • Numerical illustrations • Summary and future work

3. Motivation • Financial aspects of planning in clinical trials or clinical development programs. • Type I error rate – Set by regulatory agencies • Type II error rate – Arbitrarily set power at 80 ~ 90%— Financial/economical justification? • Comparison among various trial designs, or design parameters (timing/number of interim analyses, power etc.)

4. Past Work • Senn (1996), Statistics in Medicine • Burman&Senn (2003), Pharmaceutical Statistics • Thach & Fisher (2002), Biometrics • Liu, Anderson, & Pledger (2003). Pearson Index Ratio of expected net profit and expected cost Resulting in relatively low power (In some cases, Power<0.6, according to Liu, Q., 2004)

5. Benefit Risk Evaluation • First consider two-stage designs— Two-stage group sequential design— Two-stage adaptive design (using fixed effect size)— Two-stage adaptive design with 2nd stage redesigned using another 2-stage adaptive design (using fixed effect size).

6. Two-Stage Design • H0: 0, vs. H1: >0 • First stage: after n1 subjects — Reject H0 if — Stop without reject H0 if— Proceed to Stage 2 with additional n2 subjects if • n2 could be fixed (group sequential) or adaptive.

7. Two-Stage Design • Various methodologies have been proposed to choose n2, e.g.,— Proschan & Hunsberger (1995)— Cui, Hung, & Wang (1999)— Liu & Chi (2001)

8. Two-Stage Design • Framework of Benefit/Risk Evaluation— Prior probability of success (reject H0).— For any given sample size n, let S(n) = Revenue generated if successful C(n) = Cost of trial — Define Benefit, B = expected net profit Risk, R = expected cost

9. Two Stage Design • Therefore, for a two-stage design Here, P1()=P[early stop for efficacy] Q1()=P[early stop for futility]P2(| p1)=P[Reject H0 at Stage 2, given p1]

10. Two-Stage Design • After incorporating prior probability of success, • These two quantities can then be used for design and comparison purposes.

11. Multi-Stage Designs • These formulae for two-stage designs can then be modified easily for multi-stage group sequential or adaptive designs • Need to be careful in computational details.

12. Benefit-Risk Evaluation Larger expected sample size  Higher cost  Longer time to market  Shorter marketing exclusivity  Less revenue by patent expiration

13. Benefit-Risk Evaluation • Efficiency: Criterion based on expected sample size. Higher efficiency = Lower expected sample size. • Effectiveness: Criterion based on benefit and risk evaluation. More effective design has higher benefit and and higher benefit risk ratio.

14. Benefit-Risk Evaluation Parameters Typically Required for Design • -spending function • Spending function of type II error rate. • Total Type I error rate (fixed by regulators) • Total Type II error rate (Sponsor’s risk), OR • Sample size

15. Benefit-Risk Evaluation We specify the following design parameters • -spending function (O’Brien-Fleming is used in our numerical examples.) • Spending function of futility alpha (Similar to spending function of type II error rate. A function similar to O’Brien-Fleming is used). • Total Type I error rate (fixed by regulators) • Total futility alpha (corresponding to a certain power given effect size under alternative hypothesis).

16. Benefit-Risk Evaluation • Sample size selection:Two approaches could be used(1) To maximize benefit given these parameters. (2) (Traditional) To achieve a certain power (futility alpha needs to be specified by some criterion.)

17. Numerical Illustration • Prior prob of success: 25% • Binary endpoint • Placebo rate: 35% • For the test drug, H0: 35%, H1: 50% • Cost function: C(n)=ac+bc*n ac = $2 mil, bc = $50K

18. Numerical Illustration • Revenue function: S(n)=s[tmax-t(n)]where tmax is time remaining for marketing exclusivity, t(n) is time to complete trial. tmax =120 months • t(n)= at+bt*nat = 12 months, bt = 1/30 months • S(t)=x*t, x=$10 mil/month.

19. Numerical Illustrations • A comparison of 2-stage group sequential design and 2-stage adaptive design • Traditional design approach: Fix power at 95%, choose futility alpha to minimize expected risk Rd() • For adaptive design, Liu & Chi (2001) is used.

20. Numerical Illustrations • Comparison of 2-stage group sequential design and 2-stage adaptive design

21. Numerical Illustrations • 2-stage adaptive designs: single stage in 2nd part vs. 2-stage adaptive in 2nd part

22. Numerical Illustrations • 2-stage adaptive design with sample size adjustment using fixed effect size can be more effective than similar group sequential designs. • 2-stage adaptive design with another 2-stage AD as 2nd stage further improves effectiveness. (Could be considered as 3-stage designs.)

23. Numerical Illustrations • To evaluate multi-stage group sequential designs, the following approach is used.—For a fixed total futility alpha, choose sample size to maximize benefit—Benefit and risk profiles are then compared among various choices of Type II error rates. In our case, we select futility alpha to minimize risk.

24. Numerical Illustrations • Two stage group sequential designDesign with minimal risk occurs at futility alpha = 0.72744.Design with maximal benefit occurs at futility alpha = 0.44733.

25. Numerical Illustrations • Two-stage group sequential design

26. Numerical Illustrations • Three-stage group sequential design

27. Numerical Illustrations • Four-stage group sequential design

28. Numerical Illustrations • 2~10-stage group sequential design: benefit

29. Numerical Illustrations • 2~10-stage group sequential design: risk

30. Numerical Illustrations • Comparison of minimal-risk group sequential designs up to 10 stages

31. Numerical Illustrations • Comparison of minimal-risk group sequential designs up to 10 stages • 5-stage design seems to have the maximal benefit. (Power=94.3%) • Risk goes down with increasing number of number of analyses

32. Summary • Financial planning is a key part of pharmaceutical business. • Neyman-Pearson theory, while dealing with power of experiments, does not address the selection of power. • Financial decision-making tends to be multi-dimensional. Need to examine both benefit and risk from a portfolio perspective to optimize output.

33. Future Work • Muller & Schaefer (2001), at each interim stage, the remaining part of trial can be redesigned under certain conditions, leading to conditional and unconditional benefit risk evaluation. • Typically, significant investment has been made by the start of pivotal trial, resulting in larger value at risk. What, if any, is its impact on the design of pivotal trial.

34. Reference • Burman & Senn (2003). Examples of option values in drug development. Pharmaceutical Statistics, 2, 113-125. • Liu & Chi (2001). On sample size and inference for two-stage adaptive designs. Biometrics, 57, 172-177. • Jennison & Turnbull (1999).Group sequential methods with applications to clinical trials. Chapman and Hall, London. • Liu, Anderson, & Pledger (2003). Benefit-risk evaluation of multi-stage adaptive designs. Manuscript. • Senn (1996). Some statistical issues in project prioritization in the pharmaceutical industry. Statistics in Medicine, 15, 2689-2702. • Thach & Fisher (2002). Self-designing clinical trials to minimize expected costs. Biometrics, 58, 432-438. • Muller & Schafer (2001). Adaptive group sequential designs for clinical trials: combining the advantages of adaptive and of classical group sequential approaches. Biometrics, 57. 886-891.