Predictive Analysis of Clinical Trials

1. Predictive Analysis of Clinical Trials Richard Simon, D.Sc. Chief, Biometric Research Branch National Cancer Institute http://brb.nci.nih.gov

2. Biomarker = Biological Measurement • Early detection biomarker • Endpoint biomarker • Prognostic biomarkers • Predictive biomarkers

3. Kinds of Biomarkers • Endpoint • Measured before, during and after treatment to monitor pace of disease and treatment effect • Pharmacodynamic (phase 0-1) • Does drug hit target • Intermediate response (phase 2) • Does drug have anti-tumor effect • Surrogate for clinical outcome (phase 3)

4. Prognostic biomarkers • Measured before treatment to indicate long-term outcome for patients untreated or receiving standard treatment • May reflect both disease aggressiveness and effect of standard treatment • Used to determine who needs more intensive treatment • Predictive biomarkers • Measured before treatment to identify who will benefit from a particular treatment

5. Validation = Fitness for Intended Use

6. Single gene or protein measurement • Scalar index or classifier that summarizes contributions of multiple genes

7. Prognostic & Predictive Biomarkersin Genomic Oncology • Many cancer treatments benefit only a minority of patients to whom they are administered • Being able to predict which patients are likely to benefit can • Help patients get an effective treatment • Help control medical costs • Improve the success rate of clinical drug development

8. Biomarker Validity • Analytical validity • Measures what it’s supposed to • Reproducible and robust • Clinical validity (correlation) • It correlates with something clinically • Medical utility • Actionable resulting in patient benefit

9. Clinical Utility • Prognostic and predictive biomarkers have utility if they are actionable for informing treatment decisions in a manner that results in patient benefit

10. Clinical Utility • Biomarker benefits patient by improving treatment decisions • Identify patients who have very good prognosis on standard treatment and do not require more intensive regimens • Identify patients who are likely or unlikely to benefit from a specific regimen

11. Prognostic markers • There is an enormous published literature on prognostic markers in cancer. • Very few prognostic markers (factors) are recommended for measurement by ASCO, are approved by FDA or are reimbursed for by payers. Very few play a role in treatment decisions.

12. Pusztai et al. The Oncologist 8:252-8, 2003 • 939 articles on “prognostic markers” or “prognostic factors” in breast cancer in past 20 years • ASCO guidelines only recommend routine testing for ER, PR and HER-2 in breast cancer • “With the exception of ER or progesterone receptor expression and HER-2 gene amplification, there are no clinically useful molecular predictors of response to any form of anticancer therapy.”

13. Prognostic Factors in Oncology • Most prognostic factors are not used because they are not therapeutically relevant • Most prognostic factor studies are not conducted with an intended use clearly in mind • They use a convenience sample of patients for whom tissue is available. • Generally the patients are too heterogeneous to support therapeutically relevant conclusions • There is rarely a validation study separate from the developmental study that addresses medical utility • An analytically validated test is rarely developed

14. Prognostic factors for such a heterogeneous group of patients is not “actionable”; i.e. does not help with trteatment decision making.

16. Major problems with prognostic studies of gene expression signatures • Inadequate focus on intended use • Cases selected based on availability of specimens rather than for relevance to intended use • Heterogeneous sample of patients with mixed stages and treatments. Attempt to disentangle effects using regression modeling • Too a great a focus on which marker is prognostic or independently prognostic, not whether the marker is effective for intended use

17. If you don’t know where you are going, you might not get thereYogi Berra

18. Prognostic Biomarkers Can be Therapeutically Relevant • <10% of node negative ER+ breast cancer patients require or benefit from the cytotoxic chemotherapy that they receive

19. OncotypeDx Recurrence Score • Intended use: • Patients with node negative estrogen receptor positive breast cancer who are going to receive an anti-estrogen drug following local surgery/radiotherapy • Identify patients who have such good prognosis that they are unlikely to derive much benefit from adjuvant chemotherapy

20. Selected patients relevant for the intended use • Analyzed the data to see if the recurrence score identified a subset with such good prognosis that the absolute benefit of chemotherapy would at best be very small in absolute terms

21. Biotechnology Has Forced Biostatistics to Focus on Prediction • This has led to many exciting methodological developments • p>>n problems in which number of genes is much greater than the number of cases • And many erroneous publications • And growing pains in transitioning from an over-dependence on inference • Many of the methods and much of the conventional wisdom of statistics are based on inference problems and are not applicable to prediction problems

22. Goodness of fit is not a proper measure of predictive accuracy • Odds ratios and hazards ratios are not proper measures of prediction accuracy • Statistical significance of regression coefficients are not proper measures of predictive accuracy

24. Prediction on Simulated Null DataSimon et al. J Nat Cancer Inst 95:14, 2003 • Generation of Gene Expression Profiles • 20 specimens (Pi is the expression profile for specimen i) • Log-ratio measurements on 6000 genes • Pi ~ MVN(0, I6000) • Can we distinguish between the first 10 specimens (Class 1) and the last 10 (Class 2)? • Prediction Method • Compound covariate predictor built from the log-ratios of the 10 most differentially expressed genes.

26. Cross Validation • Cross-validation simulates the process of separately developing a model on one set of data and predicting for a test set of data not used in developing the model • The cross-validated estimate of misclassification error is an estimate of the prediction error for model fit using specified algorithm to full dataset

27. Cross-validation Estimate of Prediction Error

28. Cross validation is only valid if the test set is not used in any way in the development of the model. Using the complete set of samples to select genes violates this assumption and invalidates cross-validation. • With proper cross-validation, the model must be developed from scratch for each leave-one-out training set. This means that feature selection must be repeated for each leave-one-out training set.

29. Predictive Biomarkers • Cancers of a primary site often represent a heterogeneous group of diverse molecular entities which vary fundamentally with regard to • the oncogenic mutations that cause them • their responsiveness to specific drugs

30. Most cancer treatments benefit only a minority of patients to whom they are administered • Being able to predict who requires intensive treatment and who is likely to benefit from which treatments could • save patients from unnecessary debilitating adverse effects of treatments that they don’t need or benefit from • enhance their chance of receiving a treatment that helps them • Help control medical costs • Improve the success rate of clinical drug development

31. In most positive phase III clinical trials comparing a new treatment to control, most of the patients treated with the new treatment did not benefit. • Adjuvant breast cancer: 70% long-term disease-free survival on control. 80% disease-free survival on new treatment. 70% of patients don’t need the new treatment. Of the remaining 30%, only 1/3rd benefit.

32. Predictive Biomarkers • Estrogen receptor over-expression in breast cancer • Anti-estrogens, aromatase inhibitors • HER2 amplification in breast cancer • Trastuzumab, Lapatinib • OncotypeDx gene expression recurrence score in breast cancer • Low score for ER+ node - -> no chemotherapy • KRAS in colorectal cancer • WT KRAS = cetuximab or panitumumab • EGFR mutation in NSCLC • EGFR inhibitor • V600E mutation in BRAF of melanoma • vemurafenib • ALK translocation in NSCLC • crizotinib

33. Standard Paradigm of Broad Eligibility Phase III Clinical Trials Sometimes Leads to • Treating many patients with few benefiting • Small average treatment effects • Problematic for health care economics • Inconsistency in results among studies • False negative studies

34. The standard approach to designing phase III clinical trials is based on two assumptions • Qualitative treatment by subset interactions are unlikely • “Costs” of over-treatment are less than “costs” of under-treatment

35. Oncology therapeutics development is now focused on molecularly targeted drugs that are only expected to be effective in a subset of patients whose tumors are driven by the molecular targets • Most new cancer drugs are very expensive • the aspirin paradigm on which some current clinical trial dogma is based is a roadblock to progress

36. Subset Analysis • In the past often studied as un-focused post-hoc analyses • Numerous subsets examined • Same data used to define subsets for analysis and for comparing treatments within subsets • No control of type I error • Led to conventional wisdom • Only hypothesis generation • Only valid if overall treatment difference is significant • Only valid if there is a significant treatment by subset interaction

37. Neither current practices of subset analysis nor current practices of ignoring differences in treatment effect among patients are effective for evaluating treatments where qualitative interactions are likely or for informing labeling indications

38. Although the randomized clinical trial remains of fundamental importance for predictive genomic medicine, some of the conventional wisdom of how to design and analyze rct’s requires re-examination • The concept of doing an rct of thousands of patients to answer a single question about average treatment effect for a target population presumed homogeneous with regard to the direction of treatment efficacy in many cases no longer has an adequate scientific basis

39. How can we develop new drugs in a manner more consistent with modern tumor biology and obtain reliable information about what regimens work for what kinds of patients?

40. Development is Most Efficient When the Scientific Basis for the Clinical Trial is Strong • Having an important molecular target • Having a drug that can inhibit the target in an overwhelming proportion of tumor cells at an achievable concentration • Having a pre-treatment assay that can identify the patients for whom the molecular target is driving progression of disease

41. When the Biology is Clear • Develop a classifier that identifies the patients likely (or unlikely) to benefit from the new drug • Classifier is based on either a single gene/protein or composite score • Develop an analytically validated test • Measures what it should accurately and reproducibly • Design a focused clinical trial to evaluate effectiveness of the new treatment in test + patients

42. Off Study Control New Drug Patient Predicted Non-Responsive Develop Predictor of Response to New Drug Patient Predicted Responsive Using phase II data, develop predictor of response to new drug Targeted (Enrichment) Design

44. Evaluating the Efficiency of Targeted Design • Simon R and Maitnourim A. Evaluating the efficiency of targeted designs for randomized clinical trials. Clinical Cancer Research 10:6759-63, 2004; Correction and supplement 12:3229, 2006 • Maitnourim A and Simon R. On the efficiency of targeted clinical trials. Statistics in Medicine 24:329-339, 2005.

45. Relative efficiency of targeted design depends on • proportion of patients test positive • specificity of treatment effect for test positive patients • When less than half of patients are test positive and the drug has minimal benefit for test negative patients, the targeted design requires dramatically fewer randomized patients than the standard design in which the marker is not used

46. Two Clinical Trial Designs • Standard design • Randomized comparison of new drug E to control C without the test for screening patients • Targeted design • Test patients • Randomize only test + patients • Treatment effect D+ in test + patients • Treatment effect D- in test – patients • Proportion of patients test + is p+ • Size each design to have power 0.9 and significance level 0.05

47. RandRat = nuntargeted/ntargeted • If D-=0, RandRat = 1/ p+2 • if p+=0.5, RandRat=4 • If D-= D+/2, RandRat = 4/(p+ +1)2 • if p+=0.5, RandRat=16/9=1.77

48. Comparing T vs C on Survival or DFS5% 2-sided Significance and 90% Power

49. Hazard ratio 0.60 for test + patients • 40% reduction in hazard • Hazard ratio 1.0 for test – patients • 0% reduction in hazard • 33% of patients test positive • Hazard ratio for unselected population is • 0.33*0.60 + 0.67*1 = 0.87 • 13% reduction in hazard

50. To have 90% power for detecting 40% reduction in hazard within a biomarker positive subset • Number of events within subset = 162 • To have 90% power for detecting 13% reduction in hazard overall • Number of events = 2172