slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self PowerPoint Presentation
Download Presentation
Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self

Loading in 2 Seconds...

play fullscreen
1 / 20

Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self - PowerPoint PPT Presentation


  • 98 Views
  • Uploaded on

Statistical positivity criteria and multiplicity adjustment methods for the analysis of gated ICS assay data. Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self Statistical Center for HIV/AIDS Research and Prevention (SCHARP) HIV Vaccine Trials Network (HVTN)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self' - tekla


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Statistical positivity criteria and multiplicity adjustment methods for the analysis of gated ICS assay data

Yunda Huang, Zoe Moodie, Steve De Rosa, Steve Self

Statistical Center for HIV/AIDS Research and Prevention (SCHARP)

HIV Vaccine Trials Network (HVTN)

Fred Hutchinson Cancer Research Center

outline
Outline
  • Positivity criteria
  • Multiplicity adjustment methods
  • Simulations
  • Assay validation results
  • Summary
positivity criteria
Positivity criteria
  • For each combination of peptide-pool stimulation, T-cell subset and cytokine subset, an assay result is categorized as positive or negative depending on whether or not the percentage of positive staining cells in the experimental wells is significant greater than that in the negative control wells
  • Response rates (number of vaccine recipients who have positive response) are the primary immunogenicity endpoint in most Phase I HIV-1 vaccine trials and one of the primary measurements in assessing surrogate endpoints in efficacy trials
  • Allow comparisons across labs, studies and population
positivity criteria1
Positivity criteria
  • Empirical positivity criteria: Background subtracted responses > = 0.05% and at least 3 times the background responses
    • Reasonable but no information on the control of false positive rate
    • Ignorance of multiple comparisons involved in assessment of an overall response
  • Statistical positivity criteria: multiplicity adjusted p-value < alpha
    • Optimizing sensitivity
    • False positive control of the assay
    • Accommodations of multiple comparisons
notation
Notation

For each sample, let and denote the cell counts of the kth, k =1, …, K, cytokine subset from the experimental well and negative control well, respectively. Then the joint distribution of the X’s:

where

n: total number of cells from the CD4 or CD8 T-cell subset

D: dependence structure

hypotheses
Hypotheses
  • In addition to (or instead of) the omnibus test,
  • To maximize the advantage of multi-parameter flow data, of interest are:

Often times, K= 2^(# of cytokines) -1. However,

more variables such as presence of at least one (or more) cytokines

are also of interest and can to be tested.

tests
Tests
  • Upper-tail two sample comparisons are constructed for each variable – the alternative hypotheses are that the stimulated responses are higher than the corresponding negative control responses.
  • Since some counts of events could be as small as 5, we use the Fisher’s exact test to account for the discreteness and sparseness of the data
need for multiplicity adjustment
Need for multiplicity adjustment
  • For each sample, there are at least 2^(#cytokines) tests are of interest for each T-cell subset and peptide pool combination
  • Without adjustment, it is too easy to reject null hypotheses, thereby reaching potentially erroneous conclusions
  • In the context of vaccine development and evaluation, it is more dangerous to claim a result to be true when it is not true than to fail to claim a result that is true
  • Need to use appropriate multiplicity adjustment methods to control family-wise error
multiplicity adjustment methods
Multiplicity adjustment methods

The Bonferroni method can be conservative due to

  • Ignorance of the correlations structures among the multiple raw p values (permutation of the whole vector)
  • Equally weighting of clear false nulls and true nulls (stepwise procedure)
  • Ignorance of the discreteness of the sampling distributions (taking into account the precise discrete characteristics)
multiple tests with discrete distributions westfall wolfinger 1997
Multiple tests with discrete distributions(Westfall & Wolfinger, 1997)
  • Using the distribution of the minimum of the p values
  • Assuming the p-values are independent
  • Using the Bonferroni in equality – discrete Bonferroni adjusted p-values:

, where Pi refer to the random null p values

, where

otherwise.

And, pit are the observable values of the random p value for the ith variable

multiple tests with discrete distributions
Multiple tests with discrete distributions
  • Using , the random null p values are generated by

re-sampling (permuting) the nE + nN binary data vectors, taking the first nE for the re-sampled “experimental well” data and the last nN for the re-sampled “negative control well” data.

    • By re-sampling vectors, the correlation among binary outcomes is incorporated.
    • In addition, any redundancy (e.g., induced by including the composite category (any)) is explicitly accounted for in the adjusted p values
    • The step-down fashion can be easily added to further enhance the power
multiple tests with discrete distributions1
Multiple tests with discrete distributions
  • Using the independence-assuming adjustment
  • Using discrete Bonferroni adjustment
  • pit are obtained by exactly enumerating the discrete vectors
  • While the re-sampling multiplicity method does incorporate correlations, the fantastic gains over the Bonferroni method come from incorporating discreteness.
other multiplicity adjustment methods
Other multiplicity adjustment methods
  • Tarone (1990) recognized the problem of the discretness of the sampling distribution, and proposed a modified Bonferroni method that incorporates only those variables that can possibly contribute to the multiplicity adjustment
  • Hommel and Krummenaur (1998)
definition of power in multiple testing
Definition of power in multiple testing

Assuming there are k null hypotheses Hi. Suppose some are true nulls, and all remaining hypotheses are false. Various definitions of power include:

  • The prob. of correctly rejecting at least one false null (minimal power)
  • The prob. of rejecting all false null hypotheses (complete power)
  • The prob. of rejecting exactly one false null, and
  • The prob. of rejecting a specific false null.
  • Minimal power is assessed in our simulations because the primary endpoint is defined as having a positive response in at least one comparison
simulation results
Simulation results

Sample size N = 50,000

Number of simulated data: 100

Number of permutations: 1000000

Alpha: 0.00001

qualitative validation of 8 color ics assays
Qualitative validation of 8-color ICS assays
  • Examine 20 HIV+ and 50 HIV- PBMC samples
  • Responses to any of Gag1, Gag2 or Nef peptide pool indicate a positive response
  • Statistical criteria for determining positivity: resampling-based DB multiplicity adjusted p-value <= 0.00001

HIV Status

# positive/total

%

Negative

Positive

3/50

20/20

6%

100%

Note: False positives were most likely due to SEB contaminations

pte peptide validation using 8 color ics assays
PTE peptide validation using 8-color ICS assays
  • Examine 10 HIV+ and 50 HIV- PBMC samples
  • Responses to any of Gag1, Gag2 , Nef, Pol1, pol2 or pol3 peptide pool indicate a positive response
  • Statistical criteria for determining positivity: resampling-based DB multiplicity adjusted p-value <= 0.00001

HIV Status

# positive/total

%

Negative

Positive

0/49

9/10

0%

90%

summary
Summary
  • A multivariate binomial framework is used in determination of variable-specific positivity of an ICS assay result
  • Re-sampling based multiplicity adjustment, in tandem with the discrete Bonferroni method is used to handle multiple comparisons to incorporate both correlations and discreteness of the data
  • This method allows incorporation of additional variables or collapsing of multiple variables for comparisons
  • Simulation studies have shown higher power of the this method as compared to others
  • High sensitivity and specificity of the 8-color ICS assay were observed using the current statistical positivity criteria
reference
Reference
  • Hommel, G. and Krummenauer, F. (1998). Improvements and modifications of Tarone’s multiple test procedure for discrete data. Biometric 54, 673 -681.
  • Leon, A. C. and Heo M. (2005). A comparison of multiplicity adjustment strategies for correlated binary endpoints. JBS 15: 839-55
  • Tarone, R. E. (1990). A modified Bonferroni method for discrete data. Biometrics 46, 515-22.
  • Westfall, P. H., Wolfinger, R. D. (1997). Multiple tests with discrete distributions. Amer. Stat. 51:3-8
  • Westfall, P. H., Young, S. S. (1989). P value adjustments for multiple tests in multivariate binomial models. JASA 84:780 -6
  • Westfall, P. H. and Young, S. S. (1993). Resampling-based multiple testing: Examples and Methods for P-value Adjustment, New York: John Wiley & Sons.
  • Westfall, P. H., Tobias R. D., Rom D., Wolfinger R. D. and Hochberg Y. (1999). Multiple Comparisons and multiple tests using SAS. Cary, NC: SAS Institute Inc.