Statistical issues in the validation of surrogate endpoints

1. Statistical issues in the validation of surrogate endpoints Stuart G. Baker, Sc.D. sb16i@nih.gov

2. Surrogate endpoint: definition • used to make conclusions about the effect of intervention on true endpoint • obtained sooner, at less cost, or less invasively than the true endpoint

3. Outline • Asking the right questions • Hypothesis testing • General framework for validation and application • Graphical view • Estimation (meta-analytic) • General framework for validation and application • Trial-level statistics –graphical view • Predicted effect of intervention for binary surrogate and true endpoints (NEW approach) – graphical view • Caveats

4. Asking the right questions

5. Asking the right questions • Validation trial: Both surrogate and true endpoints are observed • QUESTION: Are the conclusions about the effect of intervention on true endpoint the same when based on (i) only surrogate endpoint (ii) only the true endpoint ? • Application trial: Surrogate but not true endpoint is observed • QUESTION: What is the effect of intervention on true endpoint?

6. Hypothesis testing

7. General Framework for Hypothesis Testing If Prentice Criteria hold, valid hypothesis testing using surrogate endpoint Surrogate endpoints True endpoints Validation trial Validation Test hypothesis using surrogate endpoint New trial Surrogate endpoint Application Valid hypothesis test: H0(T): no effect of intervention on true implies H0(S): no effect of intervention on surrogate so that reject H0(S) implies reject H0(T) Prentice Criteria; pr (true | surrogate) not depend on group + extra requirement (if binary: surrogate predicts true) (Buyse,Molenbergs, 1998)): easy to reject;hard to show they hold

8. Understanding hypothesis testing • Graphical illustration using • Binary surrogate endpoint • Binary true endpoint

9. Prentice Criterion “holds” Fraction with true endpoint Treatment A Validation Trial Treatment B Fraction with surrogate endpoint 0 0 Extrapolation: Prentice Criterion “holds” (but how close is close enough?) Fraction with true endpoint B A Application trial 0 Fraction with a surrogate endpoint A B 1 0 A=B for true implies A=B for surrogate

10. Prentice Criterion does not hold Fraction with true endpoint Treatment A Validation Trial Treatment B Fraction with surrogate endpoint 0 0 Fraction with true endpoint Extrapolation: same lines Hypothesis testing gives incorrect conclusion A B Application trial Fraction with a surrogate endpoint 0 A B 1 0 A=B for true does not imply A=B for surrogate

11. Estimation Meta-analytic (based on multiple previous trials)

12. General Framework for Estimation Surrogate endpoints True endpoints Previous trials Predicted effect of intervention on true endpoint model Validation (similar confidence intervals) Surrogate endpoint Validation trial Observed effect of intervention on true endpoint True endpoint Application trial Predicted effect of intervention on true endpoint Surrogate endpoint Application

13. Meta-analytic methods of estimation • Trial-level statistics • Buyse et al (2000); Gail et al (2000) • Estimated predicted effect of intervention on true endpoint • proposal for binary surrogate and true endpoints • simple computations

14. Focus • Binary surrogate endpoint • Binary true endpoint

15. DATA SCHEME Application trial Validation trial Previous trial 1 Previous trial 2 Previous trial 3

16. Meta-analysis of trial-level statistics Graphical overview of approach of Buyse et al (2000) and Gail et al (2000)

17. Trial-level meta-analysis (simplified overview) Regression: using random effects and within trial data Previous study 1 Fraction with true endpoint Previous study 2 Previous study 3 d 0 Fraction with surrogate endpoint A B 1 0

18. Meta-analysis of estimated predicted effects of intervention A new approach for binary surrogate and true endpoints

19. Predicted effect of intervention on true endpoint based on surrogatesA and B in new study and data from previous study 1 Fraction with true endpoint d1 0 Fraction with surrogate endpoint A B 1 0 Note: Lines for each group need not be identical—Prentice Criterion not needed

20. Predicted effect of intervention on true endpoint based on surrogatesA and B in new study and data from previous study 2 Fraction with true endpoint d2 d1 0 Fraction with surrogate endpoint A B 1 0 Note: Lines for each group need not be identical—Prentice Criterion not needed

21. Predicted effect of intervention on true endpoint based on surrogatesA and B in new study and data from previous study 3 Fraction with true endpoint d2 d1 d3 0 Fraction with surrogate endpoint A B 1 0 d=(d1w1+ d2w2+ d3w3)/(w1+w2+w3 )

22. Meta-analysis of estimated predicted treatment effects: d1, d2, d3 d= (d1 w1+d2 w2 + d3 w3)/(w1+w2+w3), • Weights wi are based on a random-effects model for di (with variance s2) • simpler than a random-effects for slopes • wi = 1 / (sampling variance of di+ s2 ) • Weights minimize variance of d if di are not correlated • simplification since di‘s are correlated

23. Meta-analysis computation • d= (d1 w1+d2 w2 + d3 w3)/(w1+w2+w3), where • di={fi0ApA+fi1A(1-pA)} - {fi0BpB+fi1B(1-pB)} • Application or validation trial: fraction with surrogate endpoint= pAandpB • Previous trials: fraction with true given surrogate endpoint: fi0A, fi1A, fi0B, fi1B • wi=1/(Vi + s2 ), Vi = sampling variance of (di) • To estimate s2 • adapt method of DerSimonian and Laird for usual meta-analysis accounting for covariance among di’s due to share parameters pAandpB • To compute variance of d • Bootstrap trials and data within trials

24. Meta-analysis simulation d= (d1 w1+d2 w2 + d3 w3)/(w1+w2+w3), where • di={fi0ApA+fi1A(1-pA)} - {fi0BpB+fi1B(1-pB)} • Simulation • Generate random fi0A, fi1A, fi0B, fi1B • Generate random data for each trial • Mean squared error • Slightly smaller for meta-analysis of predicted effect of intervention than for meta-analysis of trial-level statistics (computed via method-of-moments)

25. Hypothetical data: Example 1

26. Hypothetical data: Example 2

27. Real Data (x 10) from multicenter trial in Gail et al (2000): surrogate is cholesterol level, true endpoint is artery diameter

28. Caveats

29. Caveats • Needed even if surrogate is validated with data from many previous studies • Extrapolation to a new trial • Hypothesis testing • Estimation using data from previous trials • Surrogate endpoint does not predict harms that might arise after surrogate is observed

30. When caveats are less critical • Preliminary drug development when the surrogate endpoint is used to decide on further development or definitive testing with a true endpoint • Establishing dose or timing of an intervention previously shown effective using true endpoint at a different (suboptimal?) dose or timing

31. Summary

32. Types of trials • Validation trial: • Both surrogate and true endpoint • Do you obtain the same conclusion about effect of intervention on true endpoint using (i) surrogate endpoint and (ii) true endpoint? • Application trial: • Only surrogate endpoint • What is the effect of intervention on true endpoint?

33. Hypothesis testing • Not validated if reject Prentice’s criteria • Not clear what to conclude about surrogate if cannot reject Prentice’s criteria

34. Estimation (meta-analysis) • Not need Prentice’s criteria • Meta-analysis of trial-level statistic • Applicable to all types of endpoints • Meta-analysis of estimated predicted effect of intervention on true endpoint • Binary surrogate and true endpoints • Computationally simple • Slightly smaller MSE than with meta-analysis of trial-level statistics (in simulation)