Methodological Issues in Physician-Level Measurement of Clinical Quality

1. Methodological Issues in Physician-Level Measurement of Clinical Quality Elizabeth A. McGlynn, Ph.D. June 26, 2006

2. Information About Individual Physicians’ Performance is Increasingly Sought • Health plans believe they can save money through differential payments to physicians (pay for performance) • Employers believe they can save money through increasing consumer cost-sharing (consumer directed health plans) • Medical groups believe they can negotiate higher rates or market share by demonstrating better performance (tiered networks, rate increases) • Consumers are likely to demand information on performance as the share they pay for health care increases (public release)

3. What Is Being Measured?

4. Data Sources for Measuring Quality • Available sources include: • Administrative (claims) data • Manual abstraction of medical records • Surveys of patients • Inspection of office practice • Extraction of data from electronic medical records • Board certification/Maintenance of certification • Each of these sources has strengths and weaknesses • No single source is adequate to address all questions

5. Most Existing Approaches to Measuring Physician Performance Use Claims Data • Data are readily available and impose less burden on providers • But they have some significant problems • Generally available one payer at a time • Information availability driven by the benefit package and the ways coding systems are used • Some confounding of physician practice patterns with patient behavior • Pressure to deliver answers driving widespread use of these methods

6. Current Approaches to Quality Measurement • “Leading indicators” • One measure at a time • Condition-specific aggregates/composites • Multiple measures on the same population with the same health problem • Comprehensive cross-condition measures • Patient as the unit of analysis

7. Examples of Where These Approaches Are Currently Used

8. What You Measure May Affect the Conclusions You Draw

9. Some Challenges in Measuring Physician Performance

10. PT1 PT3 PT2 PT4 MD1 MD2 PT3 PT5 Physicians See Multiple [Different] Patients So, representing the variety of practice matters: Case Mix Adjustment

11. A Market Basket of Indicators May Be Necessary to Reflect the Variety of Practice

12. PT3 PT2 PT4 MD1 PT5 MD2 PT1 PT6 MD3 PT7 PT9 PT8 Patients See Multiple Providers Hosp A Hosp B So, determining who is “responsible” matters Attribution

13. Information Rarely Available to Link Patients to Physicians a Priori • As gatekeeper models decline, no clear assignment of patients to a physician exists • Algorithms are used to “assign” patients to physicians • Done most frequently in economic profiling • Basis is majority of dollars or visits • We are experimenting with other rules: • First eligible provider seen in study period • Provider “triggering” eligibility for indicator • Critical to reality test assignments

14. Humana PacifiCare Wellpoint MD1 United MD2 Medicare Aetna MD3 Anthem Medicare Medicare Physicians Have Multiple Contracts So, putting the pieces together matters: Aggregation

15. Few Physicians Can Be Evaluated Using Single Indicators from One Payer

16. Physicians Practice in Different Systems So, understanding the organizational context matters: Fair comparisons

17. Little Routine Information Available on Physician Practice Setting • Taking organizational context into account is challenging because of data limitations • Using location may be misleading • Shared space vs. shared practice • Rationale for constructing scores at group level: • Increase sample size • Demonstrate value of integrated medical groups • Avoid scores at the physician level • Relatively little known about within vs. between group variation

18. Categorizing Physician-Level Results • Many applications of physician-level scoring require using results to categorize physicians • In/out of network • In/out of performance bonus • Tiering • We prefer statistical testing to straight cut-points • Applied this to the three different approaches to MD-level scoring • Test performance compared to the mean • Use 95% confidence interval around each provider’s score • Those with scores significantly below average were assigned to the low performance category

19. Different Methods Will Result in Different Category Assignments

20. Different Results Under Different Systems Likely To Produce Challenges from MDs

21. Summary • A number of methodological issues arise in creating quality scores at the physician level • We need to better understand the implications of these methodological choices • Because the data on which the scores are based were not intended for this purpose, feedback loops and data quality improvement are essential • But, the world isn’t going to wait for us to get the methods perfect…

22. This Train Is Headed Your Way!