Development and validation of prognostic classifiers using high dimensional data
1 / 127

Development and Validation of Prognostic Classifiers using High Dimensional Data - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Development and Validation of Prognostic Classifiers using High Dimensional Data. Richard Simon, D.Sc. Chief, Biometric Research Branch National Cancer Institute Class Prediction. Predict which tumors will respond to a particular treatment

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Development and Validation of Prognostic Classifiers using High Dimensional Data

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Development and Validation of Prognostic Classifiers using High Dimensional Data

Richard Simon, D.Sc.

Chief, Biometric Research Branch

National Cancer Institute

Class Prediction

  • Predict which tumors will respond to a particular treatment

  • Predict which patients will relapse after a particular treatment

Class Prediction

  • A set of genes is not a classifier

  • Testing whether analysis of independent data results in selection of the same set of genes is not an appropriate test of predictive accuracy of a classifier

Components of Class Prediction

  • Feature (gene) selection

    • Which genes will be included in the model

  • Select model type

    • E.g. Diagonal linear discriminant analysis, Nearest-Neighbor, …

  • Fitting parameters (regression coefficients) for model

    • Selecting value of tuning parameters

  • Estimating prediction accuracy

Class Prediction ≠ Class Comparison

  • The criteria for gene selection for class prediction and for class comparison are different

    • For class comparison false discovery rate is important

    • For class prediction, predictive accuracy is important

  • Demonstrating statistical significance of prognostic factors is not the same as demonstrating predictive accuracy.

  • Statisticians are used to inference, not prediction

  • Most statistical methods were not developed for p>>n prediction problems

Feature (Gene) Selection

  • Select genes that are differentially expressed among the classes at a significance level  (e.g. 0.01)

    • The  level is a tuning parameter which can be optimized by “inner” cross-validation

    • For class comparison false discovery rate is important

    • For class prediction, predictive accuracy is important

    • For prediction it is usually more serious to exclude an informative variable than to include some noise variables

  • L1 penalized likelihood or predictive likelihood for gene selection with penalty value selected by cross-validation

    • Select smallest number of genes consistent with accurate prediction

Optimal significance level cutoffs for gene selection. 50 differentially expressed genes out of 22,000 on n arrays

Complex Gene Selection

  • Small subset of genes which together give most accurate predictions

    • Genetic algorithms

  • Little evidence that complex feature selection is useful in microarray problems

    • Failure to compare to simpler methods

    • Improper use of cross-validation

Linear Classifiers for Two Classes

Linear Classifiers for Two Classes

  • Fisher linear discriminant analysis

  • Diagonal linear discriminant analysis (DLDA) assumes features are uncorrelated

  • Compound covariate predictor (Radmacher)

  • Golub’s weighted voting method

  • Support vector machines with inner product kernel

Fisher LDA

The Compound Covariate Predictor (CCP)

  • Motivated by J. Tukey, Controlled Clinical Trials, 1993

  • A compound covariate is built from the basic covariates (log-ratios)

    tj is the two-sample t-statistic for gene j.

    xijis the log-expression measure of sample i for gene j.

    Sum is over selected genes.

  • Threshold of classification: midpoint of the CCP means for the two classes.

Linear Classifiers for Two Classes

  • Compound covariate predictor

    Instead of for DLDA

Support Vector Machine

Other Simple Methods

  • Nearest neighbor classification

  • Nearest k-neighbors

  • Nearest centroid classification

  • Shrunken centroid classification

Nearest Neighbor Classifier

  • To classify a sample in the validation set as being in outcome class 1 or outcome class 2, determine which sample in the training set it’s gene expression profile is most similar to.

    • Similarity measure used is based on genes selected as being univariately differentially expressed between the classes

    • Correlation similarity or Euclidean distance generally used

  • Classify the sample as being in the same class as it’s nearest neighbor in the training set

When p>>n

  • It is always possible to find a set of features and a weight vector for which the classification error on the training set is zero.

  • Why consider more complex models?

  • Some predictive classification problems are easy

    • Toy problems

    • Most all methods work well

    • It is possible to predict accurately with few variables

  • Many real classification problems are difficult

    • Difficult to distinguish informative variables from the extreme order statistics of noise variables

    • Comparative studies generally indicate that simpler methods work as well or better for microarray problems because they avoid overfitting the data.

Other Methods

  • Top-scoring pairs

  • CART

  • Random Forrest

Dimension Reduction Methods

  • Principal component regression

  • Supervised principal component regression

  • Partial least squares

  • Compound covariate and DLDA

  • Supervised clustering

  • L1 penalized logistic regression

When There Are More Than 2 Classes

  • Nearest neighbor type methods

  • Decision tree of binary classifiers

Decision Tree of Binary Classifiers

  • Partition the set of classes {1,2,…,K} into two disjoint subsets S1 and S2

    • e.g. S1={1}, S2 ={2,3,4}

    • Develop a binary classifier for distinguishing the composite classes S1 and S2

      • Compute the cross-validated classification error for distinguishing S1 and S2

  • Repeat the above steps for all possible partitions in order to find the partition S1and S2 for which the cross-validated classification error is minimized

  • If S1and S2 are not singleton sets, then repeat all of the above steps separately for the classes in S1and S2 to optimally partition each of them

cDNA Microarrays

Parallel Gene Expression Analysis

Gene-Expression Profiles in Hereditary Breast Cancer

  • Breast tumors studied:

    • 7 BRCA1+ tumors

    • 8 BRCA2+ tumors

    • 7 sporadic tumors

  • Log-ratios measurements of 3226 genes for each tumor after initial data filtering


Can we distinguish BRCA1+ from BRCA1– cancers and BRCA2+ from BRCA2– cancers based solely on their gene expression profiles?



Classification of BRCA2 Germline Mutations

Evaluating a Classifier

  • “Prediction is difficult, especially the future.”

    • Neils Bohr

Evaluating a Classifier

  • Fit of a model to the same data used to develop it is no evidence of prediction accuracy for independent data

    • Goodness of fit vs prediction accuracy

  • Hazard ratios and statistical significance levels are not appropriate measures of prediction accuracy

  • A hazard ratio is a measure of association

    • Large values of HR may correspond to small improvement in prediction accuracy

  • Kaplan-Meier curves for predicted risk groups within strata defined by standard prognostic variables provide more information about improvement in prediction accuracy

  • Time dependent ROC curves within strata defined by standard prognostic factors can also be useful

Time-Dependent ROC Curve

  • Sensitivity vs 1-Specificity

  • Sensitivity = prob{M=1|S>T}

  • Specificity = prob{M=0|S<T}

Validation of a Predictor

  • Internal validation

    • Re-substitution estimate

      • Very biased

    • Split-sample validation

    • Cross-validation

  • Independent data validation

Validating a Predictive Classifier

  • Fit of a model to the same data used to develop it is no evidence of prediction accuracy for independent data

    • Goodness of fit is not prediction accuracy

  • Demonstrating statistical significance of prognostic factors is not the same as demonstrating predictive accuracy

  • Demonstrating stability of selected genes is not demonstrating predictive accuracy of a model for independent data

Split-Sample Evaluation

  • Training-set

    • Used to select features, select model type, determine parameters and cut-off thresholds

  • Test-set

    • Withheld until a single model is fully specified using the training-set.

    • Fully specified model is applied to the expression profiles in the test-set to predict class labels.

    • Number of errors is counted

    • Ideally test set data is from different centers than the training data and assayed at a different time

log-expression ratios

training set


test set

Cross-Validated Prediction (Leave-One-Out Method)

1. Full data set is divided into training and test sets (test set contains 1 specimen).

2. Prediction rule is built from scratch using the training set.

3. Rule is applied to the specimen in the test set for class prediction.

4. Process is repeated until each specimen has appeared once in the test set.

Leave-one-out Cross Validation

  • Omit sample 1

    • Develop multivariate classifier from scratch on training set with sample 1 omitted

    • Predict class for sample 1 and record whether prediction is correct

Leave-one-out Cross Validation

  • Repeat analysis for training sets with each single sample omitted one at a time

  • e = number of misclassifications determined by cross-validation

  • Subdivide e for estimation of sensitivity and specificity

  • Cross validation is only valid if the test set is not used in any way in the development of the model. Using the complete set of samples to select genes violates this assumption and invalidates cross-validation.

  • With proper cross-validation, the model must be developed from scratch for each leave-one-out training set. This means that feature selection must be repeated for each leave-one-out training set.

    • Simon R, Radmacher MD, Dobbin K, McShane LM. Pitfalls in the analysis of DNA microarray data. Journal of the National Cancer Institute 95:14-18, 2003.

  • The cross-validated estimate of misclassification error is an estimate of the prediction error for model fit using specified algorithm to full dataset

Permutation Distribution of Cross-validated Misclassification Rate of a Multivariate ClassifierRadmacher, McShane & SimonJ Comp Biol 9:505, 2002

  • Randomly permute class labels and repeat the entire cross-validation

  • Re-do for all (or 1000) random permutations of class labels

  • Permutation p value is fraction of random permutations that gave as few misclassifications as e in the real data

Prediction on Simulated Null Data

  • Generation of Gene Expression Profiles

  • 14 specimens (Pi is the expression profile for specimen i)

  • Log-ratio measurements on 6000 genes

  • Pi ~ MVN(0, I6000)

  • Can we distinguish between the first 7 specimens (Class 1) and the last 7 (Class 2)?

  • Prediction Method

  • Compound covariate prediction (discussed later)

  • Compound covariate built from the log-ratios of the 10 most differentially expressed genes.

Major Flaws Found in 40 Studies Published in 2004

  • Inadequate control of multiple comparisons in gene finding

    • 9/23 studies had unclear or inadequate methods to deal with false positives

      • 10,000 genes x .05 significance level = 500 false positives

  • Misleading report of prediction accuracy

    • 12/28 reports based on incomplete cross-validation

  • Misleading use of cluster analysis

    • 13/28 studies invalidly claimed that expression clusters based on differentially expressed genes could help distinguish clinical outcomes

  • 50% of studies contained one or more major flaws

Class Prediction

  • Cluster analysis is frequently used in publications for class prediction in a misleading way

Fallacy of Clustering Classes Based on Selected Genes

  • Even for arrays randomly distributed between classes, genes will be found that are “significantly” differentially expressed

  • With 10,000 genes measured, about 500 false positives will be differentially expressed with p < 0.05

  • Arrays in the two classes will necessarily cluster separately when using a distance measure based on genes selected to distinguish the classes


  • Split sample validation is superior to LOOCV or 10-fold CV for estimating prediction error

Comparison of Internal Validation MethodsMolinaro, Pfiffer & Simon

  • For small sample sizes, LOOCV is much less biased than split-sample validation

  • For small sample sizes, LOOCV is preferable to 10-fold or 5-fold cross-validation or repeated k-fold versions

  • For moderate sample sizes, 10-fold is preferable to LOOCV

Simulated Data40 cases, 10 genes selected from 5000

Simulated Data40 cases

Sample Size Planning References

  • K Dobbin, R Simon. Sample size determination in microarray experiments for class comparison and prognostic classification. Biostatistics 6:27, 2005

  • K Dobbin, R Simon. Sample size planning for developing classifiers using high dimensional DNA microarray data. Biostatistics 8:101, 2007

  • K Dobbin, Y Zhao, R Simon. How large a training set is needed to develop a classifier for microarray data? Clinical Cancer Res 14:108, 2008

Development of Empirical Gene Expression Based Classifier

  • 20-30 phase II responders are needed to compare to non-responders in order to develop signature for predicting response

Sample Size Planning for Classifier Development

  • The expected value (over training sets) of the probability of correct classification PCC(n) should be within  of the maximum achievable PCC()

Probability Model

  • Two classes

  • Log expression or log ratio MVN in each class with common covariance matrix

  • m differentially expressed genes

  • p-m noise genes

  • Expression of differentially expressed genes are independent of expression for noise genes

  • All differentially expressed genes have same inter-class mean difference 2

  • Common variance for differentially expressed genes and for noise genes


  • Feature selection based on univariate t-tests for differential expression at significance level 

  • Simple linear classifier with equal weights (except for sign) for all selected genes. Power for selecting each of the informative genes that are differentially expressed by mean difference 2 is 1-(n)

  • For 2 classes of equal prevalence, let 1 denote the largest eigenvalue of the covariance matrix of informative genes. Then

Sample size as a function of effect size (log-base 2 fold-changebetween classes divided by standard deviation). Two different tolerances shown, . Each class is equally represented in the population. 22000 genes on an array.

Developing and Validating Classifiers of Survival Risk

BRB-ArrayToolsSurvival Risk Group Prediction

  • No need to transform data to good vs bad outcome. Censored survival is directly analyzed

  • Gene selection based on significance in univariate Cox Proportional Hazards regression

  • Uses k principal components of selected genes

  • Gene selection re-done for each resampled training set

  • Develop k-variable Cox PH model for each leave-one-out training set

BRB-ArrayToolsSurvival Risk Group Prediction

  • Classify left out sample as above or below median risk based on model not involving that sample

  • Repeat, leaving out 1 sample at a time to obtain cross-validated risk group predictions for all cases

  • Compute Kaplan-Meier survival curves of the two predicted risk groups

  • Permutation analysis to evaluate statistical significance of separation of K-M curves

BRB-ArrayToolsSurvival Risk Group Prediction

  • Compare Kaplan-Meier curves for gene expression based classifier to that for standard clinical classifier

  • Develop classifier using standard clinical staging plus genes that add to standard staging

Does an Expression Profile Classifier Predict More Accurately Than Standard Prognostic Variables?

  • Not an issue of which variables are significant after adjusting for which others or which are independent predictors

    • Predictive accuracy and inference are different

  • The predictiveness of the expression profile classifier can be evaluated within levels of the classifier based on standard prognostic variables

Survival Risk Group Prediction

  • LOOCV loop:

    • Create training set by omitting i’th case

  • Develop PH model for training set

  • Compute predictive index for i’th case using PH model developed for training set

  • Compute percentile of predictive index for i’th case among predictive indices for cases in the training set

Survival Risk Group Prediction

  • Plot Kaplan Meier survival curves for cases with predictive index percentiles above 50% and for cases with cross-validated risk percentiles below 50%

    • Or for however many risk groups and thresholds is desired

  • Compute log-rank statistic comparing the cross-validated Kaplan Meier curves

Survival Risk Group Prediction

  • Evaluate individual genes by fitting single variable proportional hazards regression models to log expression for each gene

  • Select genes based on p-value threshold for single gene PH regressions

  • Compute first k principal components of the selected genes

  • Fit PH regression model with the k pc’s as predictors. Let b1 , …, bk denote the estimated regression coefficients

  • To predict for case with expression profile vector x, compute the k supervised pc’s y1 , …, yk and the predictive index  = b1 y1 + … + bk yk

Survival Risk Group Prediction

  • Repeat the entire procedure for permutations of survival times and censoring indicators to generate the null distribution of the log-rank statistic

    • The usual chi-square null distribution is not valid because the cross-validated risk percentiles are correlated among cases

  • Evaluate statistical significance of the association of survival and expression profiles by referring the log-rank statistic for the unpermuted data to the permutation null distribution

R. Simon, G. Bianchini, M. Zambetti, S. Govi, G. Mariani, M. L. Carcangiu, P. Valagussa, L. Gianni

National Cancer Institute, Bethesda, MD; Fondazione IRCCS - Istituto Tumori di Milano, Milan, Italy

Outcome prediction in estrogen-receptor positive, chemotherapy and tamoxifen treated patients with locally advanced breast cancer


  • Fifty-seven patients with ER positive tumors enrolled in a neoadjuvant clinical trial for LABC were evaluated. All patients had been treated with doxorubicin and paclitaxel q 3wk x 3, followed by weekly paclitaxel x 12 before surgery, then adjuvant intravenous CMF q 4wk x 4 and thereafter tamoxifen.

  • High-throughput qRT-PCR gene expression analysis in paraffin-embedded formalin-fixed core biopsies at diagnosis was performed by Genomic Health to quantify expression of 363 genes (plus 21 for Oncotype DXTM determination), as described previously (Gianni L, JCO 2005). RS genes were excluded from analysis.


  • Three models (prognostic index) were developed to predict Distant Event Free Survival (DEFS):

    • GENE MODEL Using only expression data, genes were selected based on univariate Cox analysis p value under a specific threshold significance level.

    • COVARIATES MODEL Using RS (as continuous variable), age and IBC status (covariates) a multivariate proportional hazards model was developed.

    • COMBINED MODEL Using a combination of these covariates and expression data, genes were selected which add to predicting survival over the predictive value provided by the covariates and under a specific threshold significance level.

  • Survival risk groups were constructed using the supervised principal component method implemented in BRB-ArrayTools (Bair E, Tibshirani R, PLOS Biology 2004).


  • In order to evaluate the predictive value for each model a complete Leave-One-Out Cross-Validation was used.

    • For each i-th cross-validated training set (with one case removed) a prognostic index (PI) function was created. The PI for the omitted patient is ranked relative to the PI for the i-th training set. Because the PI is a continuous variable, a cut-off percentiles have to be pre-specified for defining the risk groups. The omitted patient is placed into a risk group based on her percentile ranking. The entire procedure has been repeated using different cut-off percentiles (BRB-ArrayTools User’s Manual v3.7).


  • Statistical significance was determined by repeating the entire cross-validation process 1000 random permutations of the survival data.

    • For GENE MODEL the p value was testing the null hypothesis that there is no relation between the expression data and survival (by providing a null-distribution of the log-rank statistic)

    • For COVARIATES MODEL the p value was the parametric log-rank test statistic between risk groups

    • For COMBINED MODEL the p value addressed whether the expression data adds significantly to risk prediction compared to the covariates

RESULTSPatients characteristics at diagnosis

  • The median follow-up was 76 months (range 18-103) (by inverse Kaplan-Meier method)

  • Patients characteristics were summarized in Table 1.



OS and DEFS of all patients

Overall Survival and Distant Event Free survival – All patients

Genes selected for the GENE MODEL and COMBINED MODEL

  • The significance level for gene selection used for the identified models was p=0.005.

  • All genes included in the COMBINED MODEL were also selected in the GENE MODEL.

Cross-validated Kaplan-Meier curves for risk groups using 50th percentile cut-off










An Evaluation of Resampling Methods for Assessment of Survival Risk Prediction in High-dimensional SettingsJyothi Subramanian and Richard Simon*Biometric Research Branch National Cancer Institute,


Resampling techniques are often used to provide an initial assessment of accuracy for prognostic prediction models developed using high-dimensional genomic data with binary outcomes. Risk prediction is most important, however, in medical applications and frequently the outcome measure is a right-censored time-to-event variable such as survival. Although several methods have been developed for survival risk prediction with high-dimensional genomic data, there has been little evaluation of the use of resampling techniques for the assessment of such models. Using real and simulated datasets, we compared several resampling techniques for their ability to estimate the accuracy of risk prediction models. Our study showed that accuracy estimates for popular resampling methods such as sample splitting and leave-one-out cross-validation have a higher mean square error than for other methods. Moreover, the large variability of the split-sample and leave-one-out cross-validation may make the point estimates of accuracy obtained using these methods unreliable and hence should be interpreted carefully. A k-fold cross-validation with k = 5 or 10 was seen to provide a good balance between bias and variability for a wide range of data settings and should be more widely adopted in practice.

Cross-Validated Time Dependent ROC Curve

n = 40

n = 80

n = 160

n = 160

n = 80

n = 40

Figure 1a. Distribution of the estimated resampling and true AUC(t) for SuperPCR. (Top) High-signal data. (Bottom) Null data.

n = 40

n = 80

n = 160

n = 40

n = 160

n = 80

Figure 1b. Distribution of the estimated resampling and true AUC(t) for SuperPCR. (Top) Lung cancer data. (Bottom) DLBCLdata.

Figure 2. In the case of the null dataset, the AUC(t) is expected to be around 0.5. The number of times ÂRS(Sn) is too pessimistic (below 0.3, black bars) or too optimistic (above 0.7, light grey bars) in a total of 100 replications for the null dataset is shown. In comparison to this, the number of times Â(Sn) is below 0.3 or above 0.7 is nil. (a) n = 40 (b) n = 80 (c) n = 160.

Figure 3: The number of times ÂRS(Sn) is too pessimistic (below 0.3, black bars) or too optimistic (above 0.7, light grey bars) in a total of 100 replications for the lung cancer dataset is compared with the number of times Â(Sn) is below 0.3 or above 0.7. (a) n = 40 (b) n = 80 (c)n = 160

Figure 4: Bias in the resampling AUC(t) estimates.

Figure 5: Mean square error in the resampling AUC(t) estimates.

Contains analysis tools that I have selected as valid and useful

Analysis wizard and multiple help screens for biomedical scientists

Imports data from all platforms and major databases

Automated import of data from NCBI Gene Express Omnibus



Diagonal linear discriminant

Compound covariate

Bayesian compound covariate

Support vector machine with inner product kernel

K-nearest neighbor

L1 penalized logistic regression

Nearest centroid

Shrunken centroid (PAM)

Random forrest

Tree of binary classifiers for k-classes

Survival risk-group

Supervised pc’s

Feature selection options

Univariate t/F statistic

Hierarchical variance option

Restricted by fold effect

Univariate classification power

Recursive feature elimination

Top-scoring pairs

Validation methods



Repeated k-fold CV

.632+ bootstrap

Predictive Classifiers in BRB-ArrayTools

Selected Features of BRB-ArrayTools

  • Multivariate permutation tests for class comparison to control number and proportion of false discoveries with specified confidence level

    • Permits blocking by another variable, pairing of data, averaging of technical replicates

  • SAM

    • Fortran implementation 7X faster than R versions

  • Extensive annotation for identified genes

    • Internal annotation of NetAffx, Source, Gene Ontology, Pathway information

    • Links to annotations in genomic databases

  • Find genes correlated with quantitative factor while controlling number of proportion of false discoveries

  • Find genes correlated with censored survival while controlling number or proportion of false discoveries

    • Kaplan Meier curves

  • Analysis of variance

Thank You

  • I’ve learned things from your questions. I hope that you have too.

  • “When you’re green you’re growing, when your ripe you’re rotting.”

  • Keep learning new things and expanding your boundaries.

  • There are major opportunities for improving public health. Think big and be brave.

  • Login