Study Design Considerations for Observational Comparative Effectiveness Research

Study Design Considerations for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) www.ahrq.gov

Outline of Material This presentation will: • Provide a rationale for study design choice and describe key design features • Define start of followup • Define inclusion and exclusion criteria at start of followup • Define exposures of interest at start of followup. • Define outcome(s) of interest • Define potential confounders

Study Design Overview • Conventional designs • Cohort • Case-control • Case-cohort • Self-controlled designs • Case-crossover • Case-time-controlled • Self-controlled case series

Issues of Bias in ObservationalComparative Effectiveness Research • Exposures or treatments are not assigned, a situation which leads to challenges ensuring internal validity, that is, the absence of bias. • To ensure internal validity, treatment groups compared must have the same underlying risk for outcome within subgroups definable by measured covariates (e.g., no unmeasured confounding). • Confounding by indication leads to higher propensity for/more intensive treatment in those with the most severe disease. • With confounding by frailty, frail patients (close to death) are less likely to be treated with preventive treatments. • Ensuring a study’s internal validity is a prerequisite for its external validity or generalizability.

Study Design: Cohort Study • Cohorts are defined by their exposure at a certain point in time (i.e., baseline date) and are followed over time for the occurrence of the outcome. • Advantages: • Has a clear timeline separating potential confounders from the exposure and the exposure from the outcome • Allows estimation of actual incidence (risk or rate) • Can assess multiple outcomes • Is easy to conceptualize • Limitations: • Is inefficient for ad hoc studies when the incidence of the outcome is low

Study Design: Case-Control Study • Identifies all incident cases that develop an outcome and compares exposure history to controls • Samples controls at random from cohort members at risk for developing an outcome • Advantages: • Oversampling cases increases computational efficiency of ad hoc studies when compared with a cohort study • Can assess multiple exposures • Limitations: • Is difficult to conceptualize • Has potential for recall bias in ad hoc studies

Study Design: Case-Cohort Study • Cohorts defined as in a cohort study • Cohort members followed for incidence of outcomes • Additional information required for analysis collected for a random sample of the cohort and all cases • Increased efficiency, when compared with a full-cohort design, if additional information needs to be collected • Decreased efficiency, when compared with a nested case-control design, unless studying multiple outcomes or estimating risk

Study Design: Case-Crossover Study • Prior exposure history of cases used as the control • Removes confounding effect of measured and unmeasured characteristics that are stable over time (e.g., genetics) • Appropriate for studying acute effects of transient exposures • Advantages: • Self-controlled • Ability to assess short-term reversible effects • Ability to inform about the time window for these effects • Limitations: • Assumes constant prevalence of treatments over time • Does not allow estimation of treatment effect in a population

Study Design: Case-Time-Controlled Study • Adjusts for calendar time trends in the prevalence of treatments, which can bias the case-crossover design • Divides the case-crossover odds ratio by the equivalent odds ratio estimated in controls • Advantages: • Not dependent on assumption of no temporal changes in the prevalence of treatment • Limitations: • Need for controls adds complexity • Control for time trend can introduce confounding

Study Design: Self-Controlled Case-Series • Estimates the immediate effect of treatment in those treated at least once • Dependent on cases that have changes in treatment during a defined observation period • Advantages: • Controls for factors that are stable over time • Cohort design has the potential to increase efficiency • Well suited for rare adverse events in vaccine safety studies • Limitations: • Limited applicability in many comparative effectiveness research studies

Study Design Features • Study setting • Consideration of the study population and data source(s) • Inclusion and exclusion criteria • Should be clearly defined • Include details about the study time period • Choice of comparators • Reduces potential for confounding by comparing treatment of interest with a different treatment for the same indication or an indication with the same potential for confounding

Other Study Design Considerations • New-User Design • The conventional prevalent user design is prone to confounding and selection bias as a result of changes in treatment effects over time. • Including only new users reduces bias and confounding associated with inclusion of prevalent users. • There must be a clear starting point for followup under similar conditions of medicalization. • Immortal Time Bias • Occurs as a result of defining the exposure during the followup time rather than before followup • New-user design and use of comparator treatments reduce potential for this bias

Conclusions • Knowledge of study design options is essential to increase internal and external validity of observational comparative effectiveness research. • Biases introduced by suboptimal study design cannot usually be removed by statistical analysis. • Cohort design is preferred when data have already been collected; the validity of a nested case-control study is equivalent, given proper control selection and timing of exposures and covariates. • It is important to define the start of followup, inclusion and exclusion criteria, outcome of interest, and potential confounders at the outset.

Summary Checklist (1 of 3)

Study Design Considerations for Observational Comparative Effectiveness Research