What EHRs Can Deliver that Randomized Clinical Trials Cannot

1. What EHRs Can Deliver that Randomized Clinical Trials Cannot Retrospective studies with long-term follow-up Robert W. Grundmeier, MD July 13, 2009

2. Disclosures • No conflicts of interest • No off-label uses of commercial products will be discussed

3. Overview • Re-use of existing clinical data in electronic health records for research • The potential and challenges • Past experience with successful retrospective cohort studies in the EHR • Urinary tract infections • Asthma • The path ahead

4. Before: Paper-Based “Poetry” Fabricated Chart...Based on a true story

5. After: Detailed “Semi-Structured” EHR Template

6. The Potential:Rich and Fresh Data • Thousands of repeated observations recorded for each potential subject over time • Longitudinal health problem diagnoses • Billing diagnoses • Vital signs and measurements • Prescriptions • Immunizations • Structured preventive health visits • Laboratory and radiology data • Procedures • And many more!

7. The Potential:Large Volume of Data • 7 Years of data • 9 Subspecialty centers • 9 Subspecialty divisions • 32 Primary care sites • 300,000+ Patients • 2,500,000+ Visits • 65,000,000+ Observations

8. Case Study #1 • Antibiotics for UTI prophylaxis (Conway PH. JAMA 2007) • Pediatric Asthma Hospitalizations and the Quality of Ambulatory Care

9. Urinary Tract Infection (UTI) Study Cohort • Almost 75,000 subjects • 30 practices • 5 years of EHR data • Urine culture results from 3 laboratories • Hospital and specialty center radiology data • One Robert Wood Johnson Fellow

10. Urinary Tract Infection (UTI) Study Findings • 12% annual incidence of recurrent UTI in children with an initial UTI • Significantly higher rates in children with high grades of vesico-ureteral reflux • Antimicrobial prophylaxis… • Did not change the rate of recurrent UTI • Increased prevalence of resistant organisms in recurrent UTI from 53% to 90%!

11. Urinary Tract Infection (UTI) Study Challenges • Diagnosis codes only had “moderate” agreement with culture results • Kappa = 0.46 • Interpreting urine culture data required some natural language processing • Validation proved this approach superior to using diagnosis These kids really had a UTI

12. Urinary Tract Infection (UTI) Study Challenges • Uncertainty about whether or not we had accurately identified the first UTI • Considered using a “birth cohort” with complete data in EHR • Instead chose to review all paper charts for patients with UTI (N=775) • 91 cases excluded due to documentation of prior UTI before EHR implementation • Only 1 case considered a “false positive”

13. Case Study #2 • Antibiotics for UTI prophylaxis (Conway PH. JAMA 2007) • Pediatric Asthma Hospitalizations and the Quality of Ambulatory Care

14. Asthma Study Questions: • Does quality of asthma care affect hospitalization rate? • Are there disparities in asthma healthcare? Methods: • Almost 6,000 subjects from 5 practices • 5 years of EHR data • 24 independent variables • 1 outcome (hospitalization) • One AHRQ Contract

15. Asthma Study Preliminary Results

16. Univariate Predictors of Asthma Hospitalization • Age < 4 years • .128 vs .063 hospitalizations per subject per month • Moderate to severe persistent asthma • .075 vs .044 hospitalizations per subject per month • African American Race • .072 vs .055 hospitalizations per subject per month • Public Insurance • .073 vs .065 hospitalizations per subject per month

17. Asthma Study Challenges: Unmeasured Attributes • Marginally adequate socioeconomic status (SES) markers for retrospective studies • Public vs. private insurance is about as good as it gets • Geocoding may help • Median census tract income • Housing type

18. Bad Luck: Simultaneous QI Efforts Inseparable

19. Asthma “Misclassification across time and space” • Common conditions are coded commonly, and reasonably well • 57,820 Patients billed for asthma care • 53,824 Patients with asthma on problem list • 54,993 Patients with at least 2 albuterol prescriptions • This is EXCELLENT correlation

20. Persistent Asthma • What about persistent asthma? • 16,949 Patients billed for “persistent asthma” • 11,943 With “persistent asthma” as a problem • But… • 23,673 Patients with at least 2 inhaled corticosteroid prescriptions which implies persistent asthma • And… • Only 3,553 With persistent symptoms based on questionnaire • Huh?

21. Non-Random Misclassification By Care Location • “It is OK to compare organizations using their electronic data because everyone has the same problems with their data… the playing field is level” • Anonymous (Hospital Executive) • “Oh, really?” –Svetlana (CBMi Data Analyst)

22. Non-Random Misclassification By Care Location (Mistake in Query) Svetlana: “…And do you really think that all the players will write their queries correctly?” WRONG!

23. Non-Random Misclassification Over Time • And, the playing field changes over time • In 2004 one could have been lulled into a false sense of security over the reliability of encounter or problem list data… Actually, WE WERE!

24. Good News! Statistical Magic for Missing Data • Asthma severity is correlated with many variables available in the EHR • Frequency, type, and dose of preventive treatment • Frequency of quick relief prescriptions • Frequency of oral steroid prescriptions • Hospitalizations • We recently imputed severity for the 20% of our population that is unclassified • The results were unbelievably accurate… like magic

25. The Bottom Line • Retrospective studies can and should be done with EHR data captured for routine care • When data are suspicious or missing, look for corroborating evidence • You don’t know what you don’t know, until you read the charts • Find cohorts enriched in the disease, brew some strong coffee, and read! • “Pound the pavement,” go to where data are collected

26. The Way Forward: Improve Data Collection This is Free! Work! Bonus! • Must think about how to make the clinician want to use the new data capture tool • We are doing a comprehensive decision support intervention regarding ear infections for this reason

27. Thank You