Establishing Efficacy through Randomized Controlled Clinical Trials

1. Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER

2. Goal of Today’s Presentation • Help you to become critical, informed readers of reports/articles from randomized controlled trials (“RCTs”), and of associated research proposals • Realize that “the devil is in the details”

3. Background Baseline health status,demographics, Change in medical intervention, lifestyle, education, Health Status = f comorbidities, other concomitant (symptom & medications, history of non-responsiveness disease metrics) to TX, host of other factors or  HS = f (HS0, MI, everything else) Issue: How to measure  HS /  MI?

4. Possibilities & Problems • Retrospective data and use of multivariate statistical analysis • Prospective “naturalistic” data and use of multivariate statistical analysis • Prospective randomized controlled (double blinded?) trial • Other?

5. Why Randomize? • Members of the treatment (“TX”) and control groups tend to be comparable on all variables, known and unknown • Provides basis for statistical analysis

6. Pitfalls to Randomization • Timing (before vs. after patient enters trial) • Keeping track of drop-outs (non-random?) • “intent to treat” analysis • “last observation carried forward” • Blinding • Randomization need not generate “median person”

7. On External Validity • What affects generalizability of study findings to population as a whole? • Exclusionary & inclusionary criteria • How to assess generalizability? • Any rigorous test? • Hawthorne effect present?

8. P-Values and Significance Levels Definitions: ObserveHypothesize MI Does Make a Difference 0+ H+ MI Does Not Make a Difference 0- H- P-Value: P (O+/H-) -- false-positive rate, often .05 or .01 -- aka Type 1 or alpha error Beta: P (0-/H+) -- false negative probability; for given n, lower alpha error rate => higher beta Sensitivity: P (0+/H+) -- true-positive rate, 1-beta or “power” Specificity: P (O-/H-) -- true-negative rate

9. Implications • P-value does not directly tell us the probability that H+ is true • Classical statisticians “reject the H-” but cannot “accept the H+” (but Bayesians can go further than classical statisticians …) • For given beta, why do “head to head” RCTs require larger sample sizes than “placebo-controlled” trials? • Difference between “statistical significance” and “practical importance”

10. Related P-Value Issues • Subgroups – must they be specified ahead of time? • Data mining and data dredging • Bonferroni adjustments to overall distinct sub-group P-values • Stopping rules • Sequential design vs. chance result • FDA “confirmatory trials”

11. Internal Validity • Does credible physiological theory suggest mechanism of action? • Persuasive studies in animals? • Replicated in other human studies? • FDA requirement for two RCTs • Study undertaken by team with good credentials? • Was publication in peer-reviewed journal?

12. Single vs. Multi-Site Trials • MS accelerates patient recruitment • Less homogeneity of patients and treatments in MS trials, but greater external validity • MS trials typically simpler • MS trials give evidence of reproducibility

13. Quality Control of RCTs • Very detailed protocol, including endpoint metrics, a hallmark of a good study – done exante, not expost • Use of “surrogate markers” / intermediate outcomes • Accuracy of case report forms • Trial design isolates effect of MI? • What if randomization “fails” in TX vs. control group on some dimensions? • Additional statistical analysis • Site effects? • Keeping track of drop-outs, and reasons for drop-outs

14. Related Class Topics • Efficacy vs. effectiveness • Use of clinical research organizations (CROs) in outsourcing clinical trials • Regulations on disseminating efficacy findings from RCTs • Use of retrospective medical claims data bases • Other outcomes (quality of life, costs, economic benefits)