BMS 617

1. BMS 617 Lecture 16: Sample size and Power Marshall University Genomics Core Facility

2. Sample Size • Virtually all the quantities we calculate from sample data depend on the size of the sample. • Small sample sizes lead to wide confidence intervals • Consequently we can fail to detect important differences even when they exist • Such experiments are called underpowered • When planning and designing experiments, we need to quantify the amount we can learn from the sample size we plan to use. Marshall University School of Medicine

3. Sample size: what not to do • It is quite common for investigators to use an ad-hoc approach to determining the sample size: • Collect some data and analyze it • If the results are not statistically significant, collect some more data and reanalyze • Repeat until the results are statistically significant. • There are major problems with this approach: • The p-values or confidence intervals from this approach cannot be interpreted • The problem is that the experiment is stopped when a statistically significant result is reached, but not stopped otherwise • It is possible that a statistically significant result would lose • significance when more data were added, but this is never tested • In theory, any hypothesis can be found to be statistically significant using this technique • Though it may take a very large number of samples Marshall University School of Medicine

4. Approaches to Sample Size Calculations • There are two valid approaches to managing sample sizes when planning experiments: • Classical: • Perform calculations before the study begins to determine the number of samples to be used. • Collect data from those samples and analyze • Adaptive: • Interim analyses are performed during the course of the study, and decisions are made to add more subjects according to the results of theses analyses Marshall University School of Medicine

5. Adaptive Experimental Designs • More sophisticated versions of the adaptive approach can modify the proportions of subjects allocated to the treatment and control groups • Can even sometimes modify the experimental protocol (e.g. dose) • The difference between this and the ad-hoc approach is that all eventualities are planned before the experiment is started • Interpretations of statistical significance take account of the adaptive process • Details are highly complex and an area of current methodological research • Beyond the scope of this course Marshall University School of Medicine

6. Simplest Example: Sample size and confidence interval for a single mean • To illustrate a sample size calculation, consider the simple problem of estimating a single mean of a population from a sample • Remember we calculated the confidence interval for the mean as (m-w, m+w) where m is the sample mean and w is a margin of error • w=t*s/√n • t* is the critical value from the t-distribution • s is the sample standard deviation • n is the sample size • If we have a margin of error we wish to target, it appears easy enough to rearrange this equation: • n=(t*s/w)2 • Note that t* depends on n, but this equation can be solved by software Marshall University School of Medicine

7. Sample Size for other confidence intervals • For the difference between two means, the best results are obtained when the two groups have equal size. • If we assume equal-sized groups, the formula for the number of subjects in each group becomes • n=2(t*s/w)2 • Here s is the pooled standard deviation, s2=s12+s22 • We can similarly flip the equations for confidence intervals for proportions: • For a single proportion, n≈4p(1-p)/w2 • For two proportions, the number in each group is n≈8p(1-p)/w2 • p is the anticipated proportion (or average of the anticipated proportions in the second case) • If p cannot be estimated, p=0.5 gives the "worst case" or largest value of n Marshall University School of Medicine

8. Practical considerations for sample size calculations • These formulae for sample size rely either on estimations of the standard deviation or the actual sample proportion • Of course, these are not known until the experiment has been performed • Which cannot be done until the sample size is determined… • In practice, the best way to estimate these is with data from a pilot experiment • If this is not available, data from similar published work can be used • But be aware that published data are more likely to have smaller variance or proportions further from 50:50 than the "average” experiment • Note also that the sample sizes calculate the number of samples that enter the analysis. You may need to increase this value to account for subjects that drop out of the study, experiments that fail, or data removed as outliers Marshall University School of Medicine

9. Sample Size and Hypothesis Testing • The most common type of sample size calculation performed is related to hypothesis testing. • It answers the question: "How many samples do I need to achieve statistical significance” • This depends on four factors: • The level of statistical significance. This is usually 0.05, but a stricter definition (say 0.01) would require more samples. • The power. Power is the chance of getting a statistically significant result if a given effect size actually exists. The higher the power required, the more samples are required. • The effect size. To achieve statistical significance with very small effect sizes required very large sample sizes. To achieve statistical significance with large effect sizes requires fewer samples. • The standard deviation (for comparing means) or actual proportion (for comparing proportions). If the SD is small, the required sample size will be smaller. For noisy data (large SD), the sample size will be larger. For proportions, the sample size required is largest when the proportion is 50%. Marshall University School of Medicine

10. Example of Sample Size Calculation Results • Sample size calculations are usually required in funding applications • The funding agency wants to know that the experiment has a reasonable chance of success • Also wants to know that the sample size is not "overkill” • i.e. that replicates are not being run unnecessarily • An example statement from a grant application might look like this: In order to determine sample size, we used the variance from a previous unpublished experiment under the same conditions in the same laboratory. The variance in log2 expression values in this experiment was 0.4468. We chose a sample size of 11 subjects per group in order to have 80% power of detecting a 1.5 fold change using a significance level of 0.05. • The values entered into a sample size calculation program were: • Significance level = 0.05, Power = 0.8, Effect size = log21.5, sd= 0.4468 Marshall University School of Medicine

11. Sample Size Calculations in Practice • In practice, the sample size is not merely constrained by the desired outcome of the experiment • Also constrained by practical considerations, such as time, funding, laboratory space, etc. • Often an investigator will perform a sample size calculation, and find that the number of required samples is impractical • Using ideal values for level of statistical significance, standard deviation, power, and effect size • It can be informative to run the sample size calculation over a range of these values • Provides information as to what can be learned with a practical sample size • May come to the conclusion that the experiment is impossible with the number of subjects • May learn that the experiment can be performed, but with a limited statistical power or with the ability only to detect large effect sizes Marshall University School of Medicine

12. Sample Sizes for Other Analyses Sample size calculations even for simple analyses are complex For more complicated analyses they may be prohibitively complex, or simply not computable For these we resort to some standard rules of thumb Marshall University School of Medicine

13. Sample Sizes for Non-parametric tests • Sample size calculations can be performed for non-parametric tests, but only if assumptions are made about the distribution • In this case, it is better to use a parametric test anyway • As long as the sample sizes are reasonably large (a few dozen), a good rule of thumb is: • Compute the sample size for the corresponding parametric test, and add 15% Marshall University School of Medicine

14. Sample Sizes for Multiple Regression • Some exact calculations are possible for multiple (and logistic) regression, but these depend on values it is extremely difficult (or impossible) to estimate • For example, the degree of correlation between the variables • In general, you need many more subjects than variables • At least ten times as many • Preferably twenty times as many • And double if you plan to use variable selection • The number of variables includes all variables used in variable selection, even if they don't appear in the final model • For logistic regression, the number of subjects is the number falling into the smallest outcome category Marshall University School of Medicine

15. Summary Sample size calculations are an essential part of experimental design and planning In their simplest form, sample size calculations merely involve rearranging equations for confidence intervals However, they typically require estimates of values which are not really known until the experiment is complete Computing sample sizes over a range of realistic values is often informative For complex analyses, use standard rules of thumb if sample size calculations are not available Marshall University School of Medicine