Loading in 2 Seconds...

Grant Stephen: Chair of the MBC Life Science Informatics Group & CEO, Tessella Inc: grant.stephen@tessella.com

Loading in 2 Seconds...

- 135 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about 'Grant Stephen: Chair of the MBC Life Science Informatics Group & CEO, Tessella Inc: grant.stephen@tessella.com' - gefjun

Download Now**An Image/Link below is provided (as is) to download presentation**
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

Download Now

Presentation Transcript

### The weight of medical knowledge

### Conceptual (scientific) Heterogeneity

### Statistical Heterogeneity

Life Science Informatics Group

Introduction to Meta-Analysis

Christopher H. Schmid, PhD

Tufts-New England Medical Center

6 June 2008

Grant Stephen: Chair of the MBC Life Science Informatics Group

& CEO, Tessella Inc: grant.stephen@tessella.com

Creating Insight & Understanding from Scientific Data

Weight of the Index Medicus According to 10-Year Periods from 1879 to 1977

Introduction

- Medicine requires hard evidence, often clinical trials
- Evidence inconsistent across studies

Different study populations

Different treatments or protocols

Quality of technical design or execution

Random variation

Reasons for combining data

- Get an overall estimate of treatment effect
- Appreciate the degree of uncertainty
- Appreciate heterogeneity
- Forces you to think rigorously about the data

Meta-Analysis

- A scientific discipline which applies a protocol to critically evaluate and uses statistical methods to combine the results of (previous) research
- Provides a quantitative summary of the overall treatment effect (typically an overall estimate and confidence interval)
- Increasingly used to understand differences among studies to explain discrepancies of results and to generate hypotheses of interactions
- To guide future research

Some terms used for meta-analysis

- Systematic Review
- Overview
- Quantitative overview
- Research (evidence) synthesis
- Research Integration
- Pooling (implies lumping data altogether)
- Combining (implies performing procedures on data)

No one will criticize you for doing a systematic review. But as soon as you combine data, you will most likely have controversy

Why is that?

- Apples and oranges (heterogeneity)
- Garbage in, garbage out (quality)
- Selection of outcomes (soft or hard)
- Selection of studies
- Publication bias
- Assumptions used in quantifying results

Characteristics of Meta-Analyses

- More heterogeneity than multicenter trial
- Meta-analysis addresses random variation
- Gives pooled estimate of treatment effect
- Can be confirmatory or exploratory
- Pooling may not be best solution
- Need to explain variation

Systematic review protocol

- Well-focused study question
- Identification of studies (design, source, search strategy)
- Eligibility criteria (study, patient, and disease characteristics, treatments, outcomes)
- Data extraction (definition of outcomes, quality assessment)
- Data summary and analysis (outcomes, intention to treat)

Issues in formulating a question

- Narrow versus broad (for individuals/ subgroups or entire population)
- Scientifically meaningful and useful (based on sound biological and epidemiological principles)
- Very broadly defined questions may be criticized for mixing apples and oranges
- Narrowly focused questions have limited generalizability and may lead to biased conclusions

Literature processing

- Questions
- Search strategy
- Screening of abstracts
- Retrieve potential articles
- Screen full articles
- Data extraction on qualifying articles

Issues in Finding and Retrieving Evidence

- Search strategies
- Sources
- Language selection
- Published vs. unpublished literature
- Use of abstracts
- File drawer problem (publication bias)
- Multiple publications on same subjects
- Disproportionate amount of data for topic

Types of Multiple Publications

- Overlapping data (preliminary and later reports)
- Same data but different authors
- Similar data (same authors) but different cohort
- need to verify with authors

Most meta-analyses are retrospective exercises, suffering from all the problems of being an observational design. We cannot fix bad data.

Basic principles in combining data

- For each analysis, one study should contribute only one effect
- Effect may be single outcome or composite of several outcomes
- Effect being combined should be same or similar across studies

What kinds of control?

- No treatment control
- Placebo
- Active comparator

Types of data that could be combined

- dichotomous (events, e.g. deaths)
- measures (odds ratios, correlations)
- continuous data (mmHg, pain scores)
- survival curves
- diagnostic test (sensitivity, specificity)
- individual patient data
- “effect size”

Issues in choosing method to combine studies

- Metrics
- Fixed vs. random effects model
- Treatment effect heterogeneity
- Baseline rate heterogeneity
- Weight

Heterogeneity (diversity)

- Is it reasonable (are studies and effects sufficiently similar) to estimate an average effect?
- Types of heterogeneity
- Conceptual (clinical) heterogeneity
- Statistical heterogeneity

Are the studies of similar treatments, populations, settings, design, etc., such that an average effect would be scientifically meaningful?

Is the observed variability of effects greater than that expected by chance alone?

Meta-analyticapproaches

- Summary point estimation (random or fixed effect model)
- Meta-regression - modeling aggregate data heterogeneity
- Baseline risk meta-regression
- Response surface - individual patients’ data analysis

Sensitivity analyses

- Exclude studies
- Analyze subgroups
- Change assumptions
- Use different metric
- Compare fixed versus random effects model
- Perform cumulative meta-analysis

Dichotomous outcomes

- Binary outcomes, event or no event, yes or no
- Most common type of outcome reported in clinical trials (about 70%)
- Examples: dead/alive, stroke/no stroke, cure/failure
- 2x2 tables commonly used to report their results
- Sometimes continuous variables are forced into dichotomous outcomes.
- E.g., a threshold could be used for pain scores and reported as improved or not improved.

Available metrics for combining dichotomous outcome data

- Odds ratio (OR)
- Risk ratio (RR)
- Risk difference (RD)
- NNT (Number Needed to Treat) can be derived (inverse of the combined risk difference) = 1/RD

Calculating treatment effects in ISIS-2

TR = 791/8592 = 0.0921

CR = 1029 / 8595 = 0.1197

RR = 0.0921 / 0.1197 = 0.77

OR = (791 x 7566) / (1029 x 7801) = 0.75

RD = 0.0921 – 0.1197 = -0.028

ISIS-2 vascular death estimate & 95% CI

Streptokinase vs. Placebo

Same change in one scale may have different meaning in another scale

Beta-Blockers after Myocardial Infarction - Secondary Prevention

Experiment Control Odds 95% CI

N Study Year Obs Tot Obs Tot Ratio Low High

=== ============ ==== ====== ====== ====== ====== ===== ===== =====

1 Reynolds 1972 3 38 3 39 1.03 0.19 5.45

2 Wilhelmsson 1974 7 114 14 116 0.48 0.18 1.23

3 Ahlmark 1974 5 69 11 93 0.58 0.19 1.76

4 Multctr. Int 1977 102 1533 127 1520 0.78 0.60 1.03

5 Baber 1980 28 355 27 365 1.07 0.62 1.86

6 Rehnqvist 1980 4 59 6 52 0.56 0.15 2.10

7 Norweg.Multr 1981 98 945 152 939 0.60 0.46 0.79

8 Taylor 1982 60 632 48 471 0.92 0.62 1.38

9 BHAT 1982 138 1916 188 1921 0.72 0.57 0.90

10 Julian 1982 64 873 52 583 0.81 0.55 1.18

11 Hansteen 1982 25 278 37 282 0.65 0.38 1.12

12 Manger Cats 1983 9 291 16 293 0.55 0.24 1.27

13 Rehnqvist 1983 25 154 31 147 0.73 0.40 1.30

14 ASPS 1983 45 263 47 266 0.96 0.61 1.51

15 EIS 1984 57 858 45 883 1.33 0.89 1.98

16 LITRG 1987 86 1195 93 1200 0.92 0.68 1.25

17 Herlitz 1988 169 698 179 697 0.92 0.73 1.18

General Weighted Average Effect Size

where: di = effect size of the ith study

wi = weight of the ith study

k = number of studies

Calculation of weights

- Generally the inverse of the variance of treatment effect
- Different formula for odds ratio, risk ratio, risk difference
- Readily available in books and software

Effect Size

- Dimensionless metric
- Combine standard deviations of diverse types of related effects
- Availability and selection of reported effects may be biased
- Variable importance of different effects
- Frequently used in education, social science literature
- Difficult to interpret results

Statistical Models of Pooling 2x2 Tables

- Fixed Effect Model
- Random Effect Model

David Bowers. Statistics from scratch. An introduction for Health Care Professionals. John Wiley & Sons, 1996.

Container with fixed number of white and black balls(fixed effects model)

Random sampling from container with fixed number of white and black balls (different sample size)

Summary point estimation

Principle - common truth

Main Model - fixed effects weighted average

Advantages - easy to interpret, applies to whole population

Disadvantages - often simplistic; not applicable with heterogeneity

Different containers with different proportions of white and black balls(Random effects model)

Random sampling from containers to get overall estimate of Ratio of white and black balls

Summary point estimation

- Principle - range of truth
- Main Model - random effects weighted average
- Advantages – realistic; allows between-study heterogeneity
- Disadvantages - less intuitive; no insight into heterogeneity

Chi-Square Homogeneity Test

NOTE: d = ln(ORi d+ = ln(ORMH) wi = 1/variance (ORi)

Variance (ORi) = 1/ai + 1/bi + 1/ci + 1/di

Differences in fixed effects and random effects weights:Magnesium for AMI

Findings of Cumulative Meta-analysis

- Clinical experts’ recommendations often are unreliably synchronized with developing RCT evidence.
- Large clinical trials often echo findings from meta-analyses of several smaller studies.
- Trends established by cumulative meta-analyses of previous studies are unlikely to be reversed.

- Model parameters are random variables, not unknown constants
- Probability distribution quantifies this uncertainty
- Prior before collecting data
- Posterior after observing data
- Likelihood is probability of observing data under model
- Statements about probability of hypotheses and parameter values
- Obtained by combining prior and likelihood with Bayes rule

- Quantify knowledge of parameters
- Incorporate all sources of variation in single model
- Inference not restricted just to model parameters
- Can be generalized to nonnormal priors and likelihoods
- (e.g. binomial)
- Posterior densities need not be normally distributed
- like maximum likelihood estimates

- Infamous because random effects and fixed effects analysis lead to different conclusions
- Random effects OR = 0.59
- Fixed effects OR = 1.02
- Very large, influential clinical trial showed treatment gave no benefit
- Contradicted earlier MA with large trial showing significant benefit

Meta-analysis for Magnesium Studies

- Pooled Odds Ratio
- Mortality Observed Posterior
- Study Treated Control Est 95% PI Est 95% PI Pr(OR<1)
- Morton 1/40 2/36 0.44 0.0, 5.0 0.54 0.2, 1.6 0.89
- Smith 2/200 7/200 0.28 0.1, 1.5 0.46 0.1, 1.1 0.96
- Abraham 1/48 1/46 0.96 0.1, 15.8 0.61 0.2, 1.9 0.84
- Feldstedt 10/150 8/148 1.25 0.5, 3.3 0.86 0.4, 1.9 0.70
- Rasmussen 9/135 23/135 0.35 0.2, 0.8 0.43 0.2, 0.9 0.99
- Ceremuz. 1/25 3/23 0.28 0.0, 2.9 0.49 0.1, 1.4 0.92
- Shechter I 1/59 9/56 0.09 0.0, 0.7 0.38 0.1, 1.4 0.97
- LIMIT 2 90/1159 118/1157 0.74 0.6, 1.0 0.73 0.6, 1.0 0.99
- ISIS-4 2216/29011 2103/29039 1.06 1.0, 1.1 1.06 1.0, 1.1 0.04
- Shechter II 2/89 12/80 0.13 0.0, 0.6 0.36 0.1, 0.9 0.99
- Singh 6/76 11/75 0.50 0.2, 1.4 0.54 0.2, 1.1 0.95
- Pooled 0.59 0.4, 0.9 0.55 0.3, 0.9 0.99

Indirect Comparisons of Multiple Treatments

Trial

1 A B

2 A B

3 B C

4 B C

5 A C

6 A C

7 A B C

We want to compare A vs. B

Direct evidence from trials 1, 2 and 7

Indirect evidence from trials 3, 4, 5, 6 and 7

- Regression analysis to identify correlations between treatment effects (outcomes) and covariates of interest (predictors)
- Unit of analysis is the individual study
- Correlation implies treatment interaction
- Factors may be study-level or subject-level
- Study-level factors include blinding, randomization, dosage, protocol
- Subject-level factors include age, gender, race, blood pressure
- i = 0 + 1Xi1 + 2Xi2 + … + ui

- Different study populations
- Different treatments or protocols
- Quality of technical design or execution
- Random variation

Problems with Meta-Regression

- Number of studies usually small
- Data may be unavailable (not conceived or not reported)
- Covariates pre-selected (biased?)
- Little variation in range of mean predictor
- Subject-level factors can be affected by ecological bias
- Causality uncertain

- Principle: risk factors differ between patients
- Main model: multivariate regression of individual patient data
- Advantages
- maximum information using patient as unit of analysis
- direct interpretation for individual patient
- no reporting bias
- no ecologic bias
- Disadvantages
- data difficult to obtain and frequently unavailable
- retrieval bias
- causality uncertain
- costly

Meta-Regression vs. Individual Patient Regression

Meta-Regression Individual Patient Regression

Cost cheap expensive

Factors study patient and study

Outcomes reported updated, complete

Data Cleaning not possible reconciliation, missing data

Bias publication data retrieval, publication

Download Presentation

Connecting to Server..