Outcomes Research

1. Outcomes Research • 8 sessions of 90 minutes • Assumes basic knowledge of multi-level modeling • 4 lecturers with 4 main topics • Bindman: risk adjustment to judge Rx effectiveness • Smith-Bindman: evaluating test performance • Osmond : multi-level modeling of outcomes • Barron: propensity scores as a means to do risk adjustment

2. Homework • 7 homework exercises • Grade based on homework and class participation/ no final • Some require ability to program in Stata • Homework due to Mintu Turakhia before the next class • epi211@mac.comfor questions and to submit homework

3. Professional Conduct • TICR has a prepared statement whose principles I support • Collaborating on homework is permitted; using answers from a student in a previous year is not • Submit your own homework using your own words • Be prepared for possibility of describing homework answers in class

4. Disclaimer • Examples draw from cardiology literature • This doesn’t represent a clinical bias so much as it reflects that state of the research • Don’t hesitate to ask if something unclear about the clinical examples • Bring up examples important to you from your own field

5. Data for Outcomes Research Andy Bindman MD Department of Medicine, Epidemiology and Biostatistics

6. What is Outcomes Research • Studies of the quality of care as judged by patients’ outcomes • IOM domains of quality • Effectiveness • Safety • Timeliness • Equity • Efficiency • Patient-Centered

7. Donabedian Model of Quality Structure Process Outcome

8. Donabedian Model of Quality Structure Process Outcome Number of nurses per hospital bed Physicians per capita

9. Donabedian Model of Quality Structure Process Outcome Beta blocker following MI Immunizations

10. Donabedian Model of Quality Structure Process Outcome Survival Functional status Satisfaction

11. Which is Best to Monitor Quality? • Structure - necessary but not sufficient • Process - many things we do/recommend don’t have proven health benefit • Outcomes - our ultimate responsibility but related to more than just the care we provide

12. Predictors of Outcomes • Outcomes = intrinsic patient risk factors treatment effectiveness quality of care random chance

13. Goals of Risk-Adjustment • Account for intrinsic patient risk factors before making inferences about effectiveness, efficiency, or quality of care • Minimize confounding bias due to nonrandom assignment of patients to different providers or systems of care

14. How is Risk Adjustment Done • On large datasets • Uses measured differences in compared groups • Model impact of measured differences between groups on variables shown, known, or thought to be predictive of outcome so as to isolate effect of predictor variable of interest

15. When Risk-Adjustment May Be Inappropriate • Processes of care which virtually every patient should receive (e.g., immunizations, discharge instructions) • Adverse outcomes which virtually no patient should experience (e.g., incorrect amputation) • Nearly certain outcomes (e.g., death in a patient with prolonged CPR in the field) • Too few adverse outcomes per provider

16. When Risk-Adjustment May Be Unnecessary • If inclusion and exclusion criteria can adequately adjust for differences • If assignment of patients is random or quasi-random

17. When Risk-Adjustment May Be Impossible • If selection bias is an overwhelming problem • If outcomes are missing or unknown for a large proportion of the sample • If risk factor data (predictors) are extremely unreliable, invalid, or incomplete

18. Data Sources for Risk-Adjustment • Administrative data are collected primarily for a different purpose (billing), but are commonly used for risk-adjustment • Disease registries

19. Sources of Administrative Data • Federal Government • Medicare • VA • State Government • Medicaid (Medi-Cal) • Hospital Discharge Data • Private Insurance

20. Dataset Resources • http://www.epibiostat.ucsf.edu/courses/RoadmapK12/PublicDataSetResources/ • http://base.google.com/base/search?a_n0=clinical+trials&a_y0=9&hl=en&gl=US

21. Advantages of Administrative Data • Computerized, inexpensive to obtain and use • Uniform definitions • Ongoing data monitoring and evaluation • Diagnostic coding (ICD-9-CM) guidelines • Opportunities for linkage (vital stat, cancer)

22. Administrative Hospital Discharge Data • Admission Date • Race • Discharge Date • Sex • Type of Admission • Date of Birth • Source of Admission • Zip Code • Principal Diagnosis • Patient SSN • Other Diagnoses • Total Charges • Principal Procedure and Date • Expected Source of Payment • Other Procedures and Dates • Disposition of Patient • External Cause of Injury • Pre-hospital Care and Resuscitation (DNR)

23. Disadvantages of Administrative Data • No control over data collection process • Missing key information about physiologic and functional status • Quality of diagnostic coding can vary across sites • Non capture of out of plan/out of hospital/out of state events

24. Linking Administrative Data • Strategy for enhancing number of predictor or outcomes variables • Deterministic linkage dependent on reliable shared identifiers such as social security numbers in both datasets • Probabilistic matching of less specific variables (age, sex, race, date of birth, etc)

25. Some Routinely Available Data Linkages • California hospital discharge data and vital statistics • Example: 30 day mortality following AMI • SEER -Medicare • Example: utilization patterns for those with breast cancer • National Health Interview Survey-Medical Expenditure Panel Survey • Example: health care costs for those with self-reported chronic conditions

26. California Hospital Discharge Data and Medicaid Eligibility Files • Creates a continuous monthly record of an individual’s pattern of Medicaid enrollment • Discharge data captures all hospitalizations regardless of whether in or out of Medicaid • Have found a 3 fold increase in hospitalizations for ambulatory care sensitive conditions for those with interrupted Medicaid coverage

27. Health Plans/Delivery Systems • Health insurance claims • Inpatient, outpatient, pharmacy, diagnostics, etc • Electronic Medical Records • VA • Kaiser • SF Dept of Public Health (THREDS)

28. THREDS • ~120,000 patients per year seen in DPH clinics/SFGH • Data begin in 1996 and updated daily • Includes demographics, insurance status,visit hx, diagnostic codes, tests ordered and results, pharmacy, link to death registry • http://ctsi.ucsf.edu/bi/threds.php

29. Disease Registries • Attempt to capture all or large sample of the cases of a specified condition • Often include more clinical information than administrative datasets • Many of these can support assessments of survival beyond acute period • May require permission/approved protocol to access all or some of the data

30. Example Registries • UNOS:national registry of patients with end stage renal disease • SEER Cancer Registry • Coronary Artery Bypass Graft Surgery: California Office of Statewide Health Planning and Development

31. Doing Your Own Risk-Adjustment vs. Using an Existing Product • Is an existing product available or affordable? • Would an existing product meet my needs? - Developed on similar patient population - Applied previously to the same condition or procedure - Data requirements match availability - Conceptual framework is plausible and appropriate - Known validity

32. Conditions Favoring Use of an Existing Product • Need to study multiple diverse conditions or procedures • Limited analytic resources • Need to benchmark performance using an external norm • Need to compare performance with other providers using the same product • Focus on resource utilization, possibly mortality

33. A Quick Survey of Existing ProductsHospital/General Inpatient • APR-DRGs (3M) • Disease Staging (SysteMetrics/MEDSTAT) • Patient Management Categories (PRI) • RAMI/RACI/RARI (HCIA) • Atlas/MedisGroups (MediQual) • Cleveland Health Quality Choice • Public domain (MMPS, CHOP, CSRS, etc.)

34. A Quick Survey of Existing ProductsIntensive Care • APACHE • MPM • SAPS • PRISM

35. A Quick Survey of Existing ProductsOutpatient Care • Resource-Based Relative Value Scale (RBRVS) • Ambulatory Patient Groups (APGs) • Physician Care Groups (PCGs) • Ambulatory Care Groups (ACGs)

36. How Do Commercial Risk-Adjustment Tools Perform • Better than age/sex to predict health care use/death • Better retrospectively (~30-50% of variation) than prospectively (~10-20% of variation) • Lack of agreement among measures • More than 20% of in-patients assigned very different severity scores depending on which tool was used (Iezzoni, Ann Intern Med, 1995)

37. Co-Morbidity or Severity? • Are patients at risk for an outcome because they have multiple conditions (co-morbidities), a more severe version of a disease (disease stage) or both? • Before adjusting for co-morbidity and or severity consider whether either is a complication of treatment (or non treatment) rather than an independent health characteristic of the patient

38. Summary • Risk adjustment is a multivariate modeling technique designed to control for patient characteristics so that judgments can be made about the quality of care • Risk adjustment requires large datasets such as administrative datasets or disease registries • Commercial risk adjustment products exist for patients in different health care settings • There are many reasons why one might choose to develop a risk adjustment model - we will talk about how to do this next week!