QIO Best Practice Methods: Inpatient Practice Survey- Distinctions about Use of Data and Analysts

AHQA Technical Conference Analyst Network Meeting March 12, 2004 QIO Best Practice Methods:Inpatient Practice Survey-Distinctions about Use of Data and Analysts

Project Team • Government Task Leader • Edwin Huff, PhD, Boston Regional Office • 5 Participating QIOs • AZ (Health Services Advisory Group) • CO (Colorado Foundation for Medical Care) • MD (Delmarva) • SC (Carolina Medical Review) • WA (Qualis Health)

QIO Best Practice Methods Special Study Origins: 5th SOW CCP top performer inquiry about how one NH hospital “produced” 100% on all CCP indicators. Purpose of onsite visit: to acknowledge high performance, and to learn, for knowledge transfer to others.

QIO Best Practice Methods Special Study Origins: Missions Accomplished with Provider: Pleasant surprise to see PRO interested in high performance, and great willingness to share details of AMI Processes of Care. Also, beginning of respectful working relationship built from inquiry and contact. Additionally, established credibility with data validity, often a challenge in initial encounters with providers.

QIO Best Practice Methods Special Study Origins: Other early work to build “What works” knowledge for accelerating improvement efforts: The “What Works” special study conducted by the Mid South QIO that focused on what hospitals did that improved QIs; The TQIP “In Search of QIO Intelligence” presented at Tri Regional. Then the Science Council made up of QIO and CMS representatives discussed QIO needs, and RFPs were drafted to pursue competitive bids on several projects including two “Best Practices” Special Studies: QIO Best Practices, Provider Best Practices.

QIO Best Practice Methods Special Study Objectives Overall, special study will address: • Are there QIO “best” practices? • What are they? • What can study QIOs learn that will be useful to the larger QIO community about trying to replicate those practices?

QIO Inpatient Practice Survey Objectives • Conduct systematic, data driven inquiry to identify “worthy” QIO practices to apply in replication test.

Survey Methods: Sample • Targeted sample based upon: • Statewide inpatient quality indicators (13) from 6th SOW • Regional representation • QIO size (large and small) • 21 QIO contractors asked to respond about 6th SOW hospital improvement activities.

Key Outcome Metric for Forming Hi/Lo QIO Groups • Whether 6th SOW re-measurement on 13 statewide inpatient QIs were each statistically significantly different from national average for that indicator • A count score based on 13 indicators common to 6th and 7th SOW +1 if indicator above average -1 if indicator below average 0 if indicator not different from average

Implicit Operational Definition of “Effective QIO/Change Agent Performance”: • An Effective Change agent is one that enables their customers to be successful.

NHQR Statistical Deviation Scoring Method Example: Beta Blockade at Discharge

State to National Comparison Example from AHRQ’s National Health Care Quality Report

Score Distribution

State Statistical Deviation Scores using 13 Inpatient QIs from 6th SOW Remeasurement Period.

Sample Groupings

Sample Group Averages • Statistical Difference Counts above or below national average (AHRQ’s NHQR reporting metric): • Hi’s: 4.9, Mid’s: 0, Lo’s -4.3(F(2,21)=71.99, P<.0001)) • Reduction in Failure Rates (contract evaluation metric): • Hi’s: 13.5%, Mid’s: 9.4%, Lo’s: 9.0% (F(1,16)=3.7, P<.07)) (Hi vs. Lo) • Median Inpatient QI (Jencks, et al ranking metric): • Hi’s: 79.6, Mid’s: 73.8, Lo’s: 69.9 (F(2,21)=3.7, P<.004))

Survey Design • Written survey with 60 questions about Creating Will for Change, Ideas for Change, and Execution of Ideas (Berwick & SC) • Questions brainstormed by community: “What do you think impacts QIO work?” • 3 staff from each QIO also answered 11 questions about organizational culture, with regard to current culture • Plus – 2-hour phone interview

Analysis of Survey • Null Hypothesis: Hi Group responses no different from Lo Group • Focused most on hi/lo groupings (8 higher scoring compared to 9 lower scoring) • Nonparametric tests of association for quantitative responses • Informal review of qualitative interview responses (immersion/crystallization)

Results: QIO internal environment Higher scoring QIOs characterized by… • Staff empowered, and informed • Teams flexible, inclusive decision-making • Staff in place & ready for start of SoW • Less staff turnover

Concerning QIO Culture, High Scoring QIOs were…

Selected Quotes about QIO Culture: Teamwork Higher Scoring: • “The project design was determined by the team.” • “It is good to have both clinical and non clinical perspectives on a team. We don’t all have to be clinical experts. … Each person’s strengths can be key. Non clinical staff can ask effective questions about process. Good to have a mix.”

Selected Quotes about QIO Culture: Teamwork Lower Scoring: “We had departments, hierarchies, silos, and compartmentalization. There were delays because of that culture.” “The decisions were driven by director level folks… We were not involved at the decision making level.”

QIOs who indicated that almost all, or more than half, of inpatient staff were in place at the beginning of the 6th SOW Lower Scoring QIOs Higher Scoring QIOs

Selected Quotes about Hiring and Turnover Lower Scoring: “We only had four [QI managers] on board initially. We had to recruit four more.” “Between the 5th and 6th SOW, we had almost 100% turnover.” “Finding people took awhile. We were not in place at the beginning of the 6th SOW.”

Interactions with Hospitals Higher scoring QIOs characterized by… • Long standing relationships with providers (institutional relationships) • Less use of QIO senior leadership during recruitment • More likely to focus on process change than data collection alone • More focus on customized, hospital-specific interactions

QIOs who reported that it was not necessary for them to recruit at all in the 6th SOW Lower Scoring QIOs Higher Scoring QIOs

Selected Quotes about Relationships with Providers Higher Scoring: “We have always worked with every hospital.” “Our one-on-one relationships with hospitals started in the 5th SOW. That’s why they don’t question working with us on projects.” Lower Scoring: “…A lot of hospital staff were not familiar with the QIO.” “Only a small proportion of hospitals had participated in QIO projects in the 5th SOW.”

Selected Quotes about Focus on Data vs. Process Higher Scoring: “We did not focus on data…Process changes were proposed and defined while we were still on-site.” “We didn’t want them to focus on data, we want them to focus on organizational change.” Lower Scoring: “We were very focused on data itself – whether they were monitoring at all – initially.” “Data analyst involvement was mainly centered around developing and maintaining a data abstraction tool for hospitals… They were not involved in day-to-day intervention planning and discussion of how to evaluate our work.”

Selected Quotes about Interactive, Customized Interventions Higher Scoring: “If a gap in the process was identified, we would work with them to tailor an intervention specific to them.” “We make the diagnosis first, then suggest a specific therapy… Each one is different, adjusted to what is.” Lower Scoring: “We treated them all the same. In the 6th SOW, the visits were cookie cutter.” “The tools were good, but what we missed was working with them to adapt the tools to their setting.”

Concerning Rich Use of Data and other Information to Support Customized interaction (not cookie cutter) High Scoring QIOs were:

Selected Quotes Concerning Rich Use of Data and other Information to Support Customized interaction: High Scoring QIO: “The QIO brought hospital specific data. The data showed the hospital, their peer group rate, the state rate, the ABC TM, and the national rate. The data report was linked to factors from the self-assessment. “You said you have XX. Hospitals who are successful tend to have XX, YY, and ZZ. Let’s talk about YY and ZZ” The data included an achievable benchmark of care for each indicator. (You take all the hospitals in the state, put results in descending order. Then you count down from the top until you have selected the hospitals that represent 10% of the population. The achievable benchmark of careTM is the rate calculated from these hospitals. Hospitals received a report explaining this calculation at baseline and at re-measurement.)”

Selected Quotes Concerning Rich Use of Data and other Information to Support Customized interaction: Lower Scoring QIOs: QIO: • “They all received the reports at the same time. … All we did was compare them [individual data] to the state. We also did rank reports, by indicator.” • QIO: • “We took them the baseline, re-measurement, and the CDAC interim sample… But they didn’t believe the data. It was a weak case – not motivating.” • QIO: • “We went out and gave them the data we had from the baseline and also the total number of Medicare discharges they had, by topic….We sent them a cluster report, comparing hospitals by size. They reported to us that they didn’t know what to do with it….We did not request ad hoc reports from the CDAC.” • QIO: • “We sent feedback reports with their individual data and statewide data.”

Selected Quotes Concerning Role of Analytic Staff: Higher Scoring QIOs: QIO: “Most of the inpatient team attended each visit (one clinical project coordinator, one non-clinical project coordinator, one analyst, and one clinical coordinator who was a physician).” QIO:“Analytic staff had a really big role. They created all the graphs, showing peer groups, etc. that we gave to hospitals at the regional meetings. Internally, they gave us updates on the data and how CMS was structuring the re-measurement. They kept the team focused on how we were doing with the data in comparison to other QIOs. In terms of presenting the data at meetings, sometimes analyst did, sometimes [name of project manager], sometimes clinical staff.” QIO: “We had a biostatistician on the provider group. The measurement group provided analysis of the quarterly data from the CDAC, evaluation strategies. They helped us look at here’s what we’ve got, based on the data, so what do we do?, etc. … The analysts helped us prepare data for regional meetings, etc.” • QIO: “We tend to have an analyst as our project manager.”

Selected Quotes Concerning Role of Analytic Staff: Lower Scoring QIOs: QIO: “Our analysts were focused on implementing the MedQuest tool. They were not as involved in day-to-day intervention planning and discussion of how to evaluate our work. They might have helped us with these issues. They were involved in data abstraction also. But no role in IQC. This could have been better.” QIO: “The analysts did regional breakdowns of the data, looking at volumes of Medicare beneficiaries. They helped us identify hot spots, target zones. They also helped prepare reports of the data, like for the letter to physicians. They did not assist with the development of intervention tools.” QIO: Regarding the role of the analysts, “They were not really active on the team in the 6th SOW; not as big a role as in the 7th. … They would come to the table when we had an idea that required an analyst, but they only came to the table if we thought we needed them for something.”

Selected Quotes Concerning Customizing Interventions Higher Scoring QIOs: QIO: “If a gap in the process was identified, we would work with them to tailor an intervention specific to them….We did tailor intervention activities to each hospital. We tried to anticipate different conditions we would find. What we did was modified to what we saw. … The intervention was tailoring…. We were responsive to the needs of the hospital….You have to understand the issues first and focus on the unique needs of the customer. The toolkits we see from other QIOs have lots of elements that might not be needed by a given hospital.”

Selected Quotes Concerning Customizing Interventions Higher Scoring QIOs: QIO: “We had a database with hospital-specific information. This included data elements about their hospital-collected data, structure of QI department, any turnover issues, qualitative notes about perceived level of physician engagement, information about specific interventions the hospital was willing to implement in the past, etc….This helped us to assess how they would take in new information… Helped us to have reasonable expectations of what a given facility could accomplish, pick our battles appropriately…It wasn’t a cookie cutter approach.”

Selected Quotes Concerning Customizing Interventions Lower Scoring QIOs: • QIO: “In the 6th SOW, we made no divisions between hospitals. We treated them all the same…. In the 6th SOW, the visits were cookie cutter. In the 7th SOW, we are creative, responsive, tailored.” • QIO: “Initially we did some mail out without follow-up. That’s not as useful.” • QIO: “The tools were good, but what we missed was working with them to adapt the tools to their setting.”

Conclusions • Effective Teamwork can have an impact on providers, through better planning, & continuity of team members and relations with providers; • Customization of project work appears to have positive benefit with hospitals; and, • Focusing on Process review & redesign, and not just data collection is important.

Next Steps:Five-State Replication Test • Disciplined effort to follow the exact same “best practice” process steps in five local environments • Choice of best practice activity informed by survey • Extensive data collected concurrently about process followed and provider reaction • Report back to QIO community exactly what happens

Phases of Replication Test • Assessment of QIO culture, building better teamwork (MD QIO is lead) • Hospital recruitment (AZ QIO) • Assessment of hospital process and use of data, leading to system change (SC QIO) • Maintaining relationships / intensity of follow-up (CO QIO) • IQC / evaluation of progress (WA QIO)

For More Information: Edwin Huff, PhD -- CMS Boston Regional Office: Ehuff@cms.hhs.gov To request copy of more detailed description of findings, contact Kathyrn Bunt -- Qualis Health kbunt@waqio.sdps.org

QIO Best Practice Methods: Inpatient Practice Survey- Distinctions about Use of Data and Analysts