Introduction to Outcomes Research Methods and Data Resources

1. Introduction to Outcomes Research Methods and Data Resources David C. Chang, PhD, MPH, MBA Director of Outcomes Research UCSD Department of Surgery

3. Surgery and public health

4. Problem in surgical clinical research • Unregulated • FDA regulation applies only to “devices” (whether a real device, or a molecular device in the form of a drug) • Procedural medicine are not regulated • Many reasons: complexity, difficulty in standardizing, difficulty of enforcement (“surgeons know best” attitude) • Self-regulation

5. Erroneous literature

6. RCTs often too late OVER EVAR-1, DREAM “Tipping Point”

7. Social responsibility It is our responsibility in academic medicine, to shoulder the responsibility that, in other fields of medicine, has been assumed by the FDA To ensure that only good treatment modalities are applied to patients

8. Biggest barrier to good research? • Not having a correctly constructed hypothesis • Incorrect design • Don’t know how to get data • Fear of statistics

9. Typical questions • Components • What/why/when/how • Verb • Condition • “Why is the sky blue?” • “What is the typical presentation of appendicitis?” • Open-ended

10. Open-ended questions • Descriptive analysis • Observational study = no comparison = no statistical test • Only one denominator • May have more than one numerator, generating more than one ratio • All ratios are calculated with the same denominator

11. Descriptive statistics P value not applicable to compare different parts of the same population

12. Value and pitfall • To explore the unknown • When you know nothing, the first step is to explore and document the numbers • Risk of over-generalizing

14. Inferential statistics P value applicable for comparing parts of two populations

15. What is a hypothesis? • Question ≠ hypothesis • Questions: usually open-ended • Hypothesis: usually is closed-ended, asking for a yes/no answer • Statistical testing can only give yes/no answers

16. The process – study design

17. Steps in constructing a hypothesis • Specify the outcomes (O in PICO) • Common oversight: Often focus on the P, but vague about O (a typical question, “What is the outcome (?) of xyz patients?”) • Specify the comparisons (C in PICO) • Not done in open-ended questions • Specify covariates (control variables, adjustment)

18. Hypothesis statement • y = b1X1 + b2X2 + b3X3 • Death = age + race + gender + insurance…

19. Inclusion/exclusion criteria • Just like a clinical trials (“eligibility criteria”) • Diagnosis and/or procedure codes? • Common mistake

20. Comparison

21. Outcome • Mortality? • Rare • Complications • Length of stay • Charges • Be judicious

22. Covariates / independent variables • Patient demographcis • Patient comorbidity • Surgeon volume • Hospital volume • Hospital type (teaching vs non-teaching) • Area (rural vs urban)

23. Hierarchy of influence on surgical outcomes Nation Region Outcomes research Hospital Surgeon Clinical trials Technique and Management Patient

24. The process – data preparation

25. Overview of public and semi-public databases Multi-specialty Specialty-specific Trauma National Trauma Databank (NTDB) Oncology Surveillance, Epidemiology, and End Results (SEER) National Cancer Databank (NCDB) Transplant United Network for Organ Sharing (UNOS) • Administrative Databases • Nationwide Inpatient Sample (NIS) • Medicare, Medicaid • California OSHPD • Clinical Databases • National Surgical Quality Improvement Program (NSQIP)

26. Administrative databases Advantages Disadvantages Limited clinical course information Limited surgical procedure information • Large patient numbers • Less selection bias • Can be linked to other databases containing other non-medical information

27. NSQIP/non-NSQIP in-hospital mortality

29. Select data elements

30. Generate new data elements • Most time consuming step of outcomes analysis • Not every component of your research question is readily available in the database • For example, comorbidity • Charlson Index, Elixhauser Index • Some common concepts actually undefined • Readmission?

31. What is a “re-admission”? • Not all “admissions” are “re-admissions” • 30-day? • Elective? • Transfers? • Diagnosis-specific? • Preventable?

32. The process – analysis

33. Hypothesis statement • y = b1X1 + b2X2 + b3X3 • Death = age + race + gender + insurance…

34. Table 1: Descriptive analysis

35. Table 2: Bi-variate analysis(unadjusted comparison)

36. Table 3: Multivariable analysis(adjusted analysis)

37. Analysis for Table 1

38. Analysis for Table 1 P value not applicable to compare different parts of the same population

39. Analysis for Table 1 • % for categorical data • Mean/median/SD for continuous data • For exploratory studies, descriptive studies, case series, etc., this would be the end of the process • Reminder, avoid overgeneralizing

41. Analysis for Table 2

42. Analysis for Table 2 • Think about data types… • Continuous data • Categorical data • (Ordinal data)

43. Analysis for Table 2 • Two questions to think about when picking a stats test… • What is my outcome/dependent variable? What is my independent/input variable? • What type of data do I have for each? • 4 possible combinations: • 2 variables • 2 data types

44. Analysis for Table 2 X = input Cat. Cont. Y = outcome Cat. c2 ROC T-test Rank sum Cont. Correlation

45. Analysis for Table 3

46. Analysis for table 3 X = input Cat. Cont. Y = outcome Logistic regression Cat. c2 ROC T-test Rank sum Linear regression Cont. Correlation

47. Subset analysis • Consistency of findings • Generalizability

48. Generalizability

50. “This is not research anymore”