Antonia Stang MDCM MBA MSc Assistant Professor University of Calgary

1. The Development of Quality Indicators for High Acuity Pediatric Conditions: Challenges in the Translation of Knowledge into Performance Measurement Antonia Stang MDCM MBA MSc Assistant Professor University of Calgary Departments of Pediatrics and Community Health Sciences

2. Disclosure • I do not have an affiliation (financial or otherwise) with any commercial organization that may have a direct or indirect connection to the content of my presentation.

3. Andre Picard, Globe and Mail, March 20, 2012 “If we want healthy citizens-as opposed to citizens who have ready access to sickness care-we need a profound philosophical shift in what we should expect from medical professionals. We need to reward and incent quality, not quantity”

4. Background Quality of Care: “the degree to which health services for individuals increases the likelihood of desired health outcomes and are consistent with current professional knowledge” (Institute of Medicine)

5. Background • Quality Indicators: explicitly defined and measurable items pertaining to the structures, processes or outcomes of care • Structures: staff, equipment, physical layout of the department, laboratory and diagnostic imaging resources • Processes: interactions between professionals and patients • Outcomes: mortality, morbidity, patient satisfaction, quality of life

6. Quality Indicator Uses • Improve health care and outcomes • Benchmark performance • Set minimum standards of care • Improve efficiency • Accountability • Transparency • Research • Pay-for-Performance

7. What Makes a Good Measure? • Impact, Opportunity, Evidence—Important to Measure and Report • Reliability and Validity—Scientific Acceptability of Measure Properties • Usability • Feasibility National Quality Forum Measure Evaluation Criteria http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx

8. Objectives • To review methods for involving stakeholders in the indicator development process. • To discuss the application of GRADE (the Grading of Recommendations Assessment, Development and Evaluation) in indicator development and selection. • To describe the challenges in developing and testing broadly applicable performance measures for high impact, relatively low frequency, conditions.

9. Study Objective • to use a systematic process involving multiple stakeholders to develop evidence based quality of care indicators for pediatric conditions requiring high acuity ED care.

10. Research Team • Antonia Stang MD MBA MSC, Principal Investigator • Astrid Guttmann MD MSc, Co-Investigator • David Johnson MD, Co-Investigator • Sharon Straus MD MSc, Co-Investigator • Lisa Hartling MSc PhD, Collaborator/methodological expert • Francois Belanger MD, Collaborator/decision maker • Angelo Mikrogianakis MD, Collaborator/decision maker • Jen Crotts RN, Research Assistant • Janie Williamson RN, Pediatric Emergency Research Team Coordinator

11. Funding • Funded by an operating grant from the Canadian Institutes of Health Research (CIHR)

12. Rationale • Lack of research on quality indicators specific to the pediatric population. • Quality measures that are part of pediatric emergency medicine practice have not been systematically validated. • Performance measures specific to pediatrics and pediatric emergency medicine have been identified as a research priority.

14. Stakeholder Involvement • Goals: • to represent different stakeholder perspectives • to incorporate scientific evidence and expert opinion

15. Stakeholder Involvement • Systematic methods to combine expert opinion and medical evidence • Consensus development conferences • Guideline based • Delphi technique • Nominal group technique • RAND/UCLA appropriateness method

16. Phase 1 Condition Selection • 32 Member advisory panel • Data on the main diagnosis for high acuity (resuscitation and emergent at triage) pediatric patients (age 0-19 yrs) seen in all EDs in Ontario and Alberta. • Criteria for Condition Selection; • importance (morbidity or mortality) • impact (potential to address gap between current and best practice) • validity (adequacy of scientific evidence linking performance of care to patient outcome)

17. Phase 1: Condition Selection Table 1: High Acuity (Resuscitation and Emergent ) Pediatric visits in 2006/2007 and 2007/2008 for all EDs in Ontario and Alberta for Selected Conditions

18. Phase 2: Indicator Development • Systematic Review of the Literature for each condition • Existing Indicators • High quality national and international guidelines (AGREE), Systematic Reviews (AMSTAR), Randomized Controlled Trials • Criteria for Indicator Development • High quality evidence linking care structure or process to patient outcome • Strongly recommended • Consistency across guidelines

19. Literature Review

20. GRADE Grading of Recommendations Assessment, Development and Evaluation 1-Very low quality: Any estimate of effect is very uncertain 2-Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate 3-Moderate Quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate 4-High Quality: Further research is very unlikely to change our confidence in the estimate of effect

22. GRADE Inter-rater Reliability *Cohen’s Kappa unweighted

23. Challenges using GRADE • Subjective • Lack of concordance between guidelines • Time consuming • Requires substantial knowledge of clinical condition and research methodology • Variable inter-rater reliability

24. Phase 3: Indicator Selection • Expert Panel Process: 2 rounds of a web-based survey and a face-to-face meeting • Indicators were selected based on two criteria rated on a scale of 1(strongly agree) to 9 (strongly disagree): • Relevance • Impact • Indicators rated ≥7 on both criteria by 70% of panelists were retained

25. Relevance Impact

27. Challenges in Indicator Development • Lack of high quality evidence on the link between treatment/processes and outcomes, particularly in the pediatric setting • Difficulty in identifying performance measures applicable to all settings

28. "The only man I know who behaves sensibly is my tailor; he takes my measurements anew each time he sees me. The rest go on with their old measurements and expect me to fit them." George Bernard Shaw

29. Phase 4:Data Collection *Based on ICD-10 code

30. Diabetic Ketoacidosis *Includes only visits with fluids or insulin started in study ED

31. Anaphylaxis * for food and insect sting induced reactions

32. Feasibility and Reliability *Number of ED visits based on ICD-10 code

33. Challenges in Indicator Testing • Feasibility and Reliability • Retrospective • Proper diagnosis is in itself a quality issue • Accuracy of ICD -10 codes • Cost/effort of data collection • Small and variable sample size

34. Lessons Learned • Allow ample time for systematic review and evidence grading • Composition of expert panel is key • Need an experienced moderator • Formal qualitative analysis of expert panel meeting • Include patient/care-giver perspective

35. Next Step • Multicentre data collection on select high acuity indicators combined with existing pediatric and emergency department performance measures • Reliability and feasibility testing • Process to outcome link

36. Campbell 2003

37. Quality Improvement and Indicator Development • Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academy Press. 2001: Washington, DC. • Campbell, S.M., et al., Research methods used in developing and applying quality indicators in primary care. BMJ, 2003. 326(7393): p. 816-9. • Chassin MR et al. Accountability Measures-Using Measurement to Promote Quality Improvement. N Eng J Med 363:7. • DonabedianA. The quality of care. How can it be assessed? JAMA. Sep 23-30 1988;260(12):1743-1748. • http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx • Center for Health Policy/Center for Primary Care and Outcomes Research & Battelle Memorial Institute. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement (Prepared by Battelle, under Contract No. 290-04-0020). Rockville, MD: Agency for Healthcare Research and Quality. May 2011.

38. Pediatric and Emergency Indicators • Alessandrini E, Gorelick MH, Shaw K, Kennebeck S. Using Performance Measures to Drive Improvement in Pediatric Emergency Care 2010; http://webcast.hrsa.gov/postevents/archivedWebcastDetail.asp?aeid=534 • BardachNS, Chien AT, Dudley A. Small Numbers Limit the Use of the Inpatient Pediatric Quality Indicators for Hospital Comparison. Academic Pediatrics, 2010. 10(4). • Guttmann A, Razzaq A, Lindsay P, Zagorski B, Anderson GM. Development of measures of the quality of emergency department care for children using a structured panel process. Pediatrics, 2006. 118(1): p. 114-23. • Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, Zwarenstein M, Anderson GM. Prioritizing performance measurement for emergency department care: consensus on evidence based quality of care indicators. CJEM 2011. 13(3):300-309.

39. Rating the Quality of Evidence • http://www.gradeworkinggroup.org/index.htm (accessed February 23, 2012) • Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE Working Group. Rating quality of evidence and strength of recommendations: What is "quality of evidence" and why is it important to clinicians? BMJ. 2008 May 3;336(7651):995-8. • Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ, for the GRADE Working Group. Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-926. • Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW Jr, Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH; GRADE Working Group. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.BMJ. 2008 May 17;336(7653):1106-10. • Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J ClinEpidemiol 2009AGREE Next Steps Consortium (2009). • The AGREE II Instrument [Electronic version]. Retrieved March 23 2012, from http://www.agreetrust.org .