OBSERVATIONAL METHODS IN COMPARATIVE EFFECTIVENESS RESEARCH John Concato, M.D., M.S., M.P.H. VA Clinical Epidemiology

2. KEY POINTS A. Understand how observational studies can provide valid results, comparable to randomized trials B. Recognize general strategies used in novel methods for conducting observational studies C. Appreciate importance of fundamental principles of patient-oriented research

3. WIDE SPECTRUM OF OBSERVATIONAL RESEARCH Key attribute • non-random, yet real-world, exposure or intervention Types of study design • cross-sectional, cohort, case-control architecture Sources of data • primary vs. secondary (“database research”) Longitudinal component • (none), prospective, retrospective

4. ROLE OF OBSERVATIONAL STUDIES Observational studies can: • provide preliminary information for randomized, controlled trials (RCTs) • assess factors that can’t be randomized (e.g., genome) • examine national-level databases or real-time problems • compliment RCTs in evaluating therapeutic effectiveness

5. PROBLEM OF CONFOUNDING Major concern regarding observational studies • occurs when extraneous factor is associated with both exposure and outcome • can cause biased results • example: possible association of alcohol and lung ca is confounded by cigarette smoking • trials are protected because randomization balances factors regarding exposure

6. ‘DOGMA’ REGARDING DESIGN Architecture Strengths Limitations Randomized trials ? ? Observational studies ? ?

7. GRADES OF EVIDENCE FOR STUDY DESIGN I At least one properly randomized, controlled trial (RCT) II-1 Well-designed trials without randomization II-2 Well-designed cohort or case-control analytic studies, preferably multi-site II-3 Multiple time series with or without the intervention, or dramatic results in uncontrolled experiments (e.g., penicillin or insulin) III Opinions of respected authorities, based on clinical experience; descriptive studies, case reports, expert committee reports U.S. Preventive Services Task Force 1996

8. RCT AS ‘GOLD STANDARD’ Comment on publication of a prominent trial: “The Heart Outcomes Prevention Evaluation (HOPE) Study helped to restrain earlier observational claims that vitamin E lowers the risk of cardiovascular disease.” New Engl J Med 2000;342:1907

9. RCTs DO NOT ALWAYS AGREE Randomized, controlled trials on same topic: • Blot, et al., 1993; ? overall mortality • [Alpha-Tocopherol] 1994; no effect on mortality • Stephens, et al., 1996; ? c.v. death, nonfatal MI • [GISSI] 1999; no benefit on death, nonfatal MI, CVA • [HOPE] 2000; no effect on c.v. outcomes

10. STRENGTHS AND LIMITATIONS OF RCTs Strengths • randomization balances prognostic factors • “prospective infrastructure” collects pertinent data Limitations • limited generalizability (e.g., restricted patient population, strict protocol) • long duration; $$$; can’t answer all questions

11. RIGOROUS ASSESSMENT OF DESIGN Architecture Strengths Limitations Randomized trials ? ? Observational studies ? ?

12. METHODOLOGICAL RESEARCH Research question: Do the results of well-designed observational studies systematically differ from results of randomized, controlled trials?

13. Conclusion: “Contrary to prevailing beliefs, results from well-designed observational studies did not systematically overestimate the magnitude of associations between exposure and outcome, compared with randomized, controlled trials.” New Engl J Med 2000; 342:1887

14. McKee, et al., BMJ 1999;319:312 “…one method does not give a consistently greater effect than the other.” Benson and Hartz, New Engl J Med 2000;342:1887 “[Observational results are] neither consistently larger than nor qualitatively different from those obtained in RCTs.” Ligthelm, et al., Clin Ther 2007;29:1284 “…observational studies can be conducted using the same exacting and rigorous standards as are used for RCTs.” VALIDITY OF OBSERVATIONAL STUDIES

15. PROPOSED FOCUS FOR EVIDENCE Architecture Strengths Limitations Randomized trials ? ? Observational studies ? ?

16. Question: Does hormone replacement therapy (HRT) protect against coronary heart disease? Overview of findings: • Observational studies: decreased coronary disease • RCTs: no benefit, or (early) harm …so are observational studies always flawed?  EXAMPLE: HORMONE REPLACEMENT 

17. COMPARISON OF RCT/OBS STUDIES FOR HRT Estimate of association from: Outcome RCT Observational Colorectal ca 0.63 (0.43-0.92) 0.66 (0.59-0.74) Hip fracture 0.66 (0.45-0.98) 0.75 (0.68-0.84) Stroke 1.41 (1.07-1.85) 1.45 (1.10-1.92) Pulm emb 2.13 (1.39-3.25) 2.1 (1.2-3.8) Coronary dz 1.29 (1.02-1.63) 0.61 (0.45-0.82)

18. ‘HOW COULD WE HAVE BEEN SO WRONG?’ “Higher socioeconomic status is associated with lower rates of cardiovascular disease and higher rates of HRT.” “Thus, rather than HRT keeping women healthy, healthy women were taking HRT.” Ann Intern Med 2002;137;290

19. OBSERV STUDIES: HRT ? CORONARY DZ

20. IMPROVED OBSERVATIONAL METHODS How can problems of confounding be avoided? • use novel methods to mimic randomization • do a better job with current observational methods

21. IMPROVED OBSERVATIONAL METHODS How can problems of confounding be avoided? • use novel methods to mimic randomization (such as propensity scores)

22. OVERVIEW OF PROPENSITY SCORES Concept: Patients with similar scores are balanced regarding baseline variables that predict treatment Format: Independent variables reduced to single term Example: Assess impact of procedure used in critical care (right heart catheterization, or RHC) on mortality

23. ‘The Effectiveness of Right Heart Catheterization in the Initial Care of Critically Ill Patients’ Methods: - calculate propensity score for receiving RHC - match patients with/without RHC based on score - determine association of RHC and mortality

24. RESULTS (WITHOUT RANDOMIZATION) 

25. PROPENSITY SCORES: CONSIDERATIONS Advantages: - focuses attention on clinical risk factors - single index used in regression/matching/stratification Disadvantages: - relies entirely on “appropriate” selection of variables - requires relatively large sample size (e.g., to match)

26. IMPROVED OBSERVATIONAL METHODS How can problems of confounding be avoided? • use novel methods to “recreate” randomization • do a better job with current observational methods

27. ‘WELL-DESIGNED’ OBSERVATIONAL STUDIES Observational (cohort) studies can replicate results of RCTs: • identify “zero-time” for patients’ eligibility and status • use specific inclusion and exclusion criteria • adjust for susceptibility (confounding) factors • use statistical methods as with RCTs Am J Med 1990;89:630

28. ‘WELL-DESIGNED’ OBSERVATIONAL STUDIES Observational (cohort) studies can replicate results of RCTs: • identify “zero-time” for patients’ eligibility and status • use specific inclusion and exclusion criteria • adjust for susceptibility (confounding) factors • use statistical methods as with RCTs Am J Med 1990;89:630

29. ‘STROBE’ CRITERIA (2007) STrengthening the Reporting of OBservational studies in Epidemiology: • international, collaborative initiative on methodology • >20 criteria applicable to (various) observational studies • published simultaneously in several journals and on web (e.g., Ann Intern Med 2007;147:W163, or on web at www.strobe-statement.org)

30. STROBE RECOMMENDATIONS Item #7 - Clearly define variables (and give criteria for): • outcomes • exposures • predictors • potential confounders • effect modifiers www.strobe-statement.org

31. STROBE RECOMMENDATIONS Item #7 - Clearly define variables (and give criteria for): • outcomes • exposures • predictors • potential confounders • effect modifiers

32. ‘SCIENCE AS EXPERIMENT; SCIENCE AS OBSERVATION’ “In a world of rigorous observational studies, expending effort to argue that one methodology is superior to another (e.g., RCTs versus observational studies) is counter-productive. The importance lies not in arguing about which method is better than the other, but what can be learned about disease activity and therapy from each type of study.” Nat Clin Pract Rheumatol 2006;2:286

33. COMPONENTS OF RESEARCH STUDY Research question: topic important?; question cogent? Study design: appropriate for question?; bias minimized? Data collection: suitable quality?; relevant to proposed study? Statistical analysis: conform to design?; understandable?

34. COMPARATIVE EFFECTIVENESS RESEARCH IN VA Research question: academic clinicians in research-friendly environment ? patient-centered topics Study design: formal planning process and stable infrastructure ? rigorous study architecture Data collection: national-level health care system and data ? comprehensive and representative information Statistical analysis: experienced and engaged biostatisticians ? relevant interpretations

35. CONCLUSIONS 1. Well-conducted observational studies can provide valid results, similar to randomized trials 2. Novel methods of observational studies (e.g., propensity scores) are useful but do not “work miracles” 3. Scientific rigor is based on pertinent research questions, suitable study designs, high-quality data, and appropriate statistical analyses