Change Starts Here .

1. Change Starts Here. The One about Outcomes and Indicators ICPC National Coordinating Center This material was prepared by CFMC (PM-4010-080 CO 2011), the Medicare Quality Improvement Organization for Colorado under contract with the Centers for Medicare & Medicaid Services (CMS), an agency of the U.S. Department of Health and Human Services. The contents presented do not necessarily reflect CMS policy.

2. Recap: measurement for IC-4 • Time series outcomes • Effect on root cause/driver • Success of the intervention • Rates; scores; rating scales • Best-fit line or other signal indicating improvement • What to do about outcomes not well portrayed as time-series • Intervention implementation • Reach/dosage of an intervention • Who was affected? • Counts • Rates among eligible population (offered, refused, completed)

3. Recap: suggested approach • Map out a detailed, community-level logic model of the intervention strategy. • Select and operationalizeoutcomes and processes from the logic model. • Develop and enforce the system for tracking implementation and outcome. • Effectively report time series data.

4. Who should participate in outcomes selection and measurement? • Considerations • Who will implement the interventions? • Who will be collecting data? • Community stakeholders • To whom will the observed changes will be meaningful? • Resources brought by the QIO • Building capacity for outcome measurement • Logic models • Data collection • Technical assistance; analytic support • Data management and reporting • Detecting improvement (tests of trend, run/control charts)

5. Recap: logic model outcomes Expected short-, medium-, and long-term changes and improvements • Short-term • Specific improvements in the targeted driver or root cause • Medium-term • Related outcomes along the causal path • Long-term • Improved care transitions • Avoided readmission • Improved health care utilization • Implications of potential negative changes or non-changes Short-/medium-term outcomes are measured for IC-4.

6. Recap: outcomes selection SMART objectives guide outcome selection. Specific Concrete; represents what, or who, is expected to change Measureable Can be seen, heard, counted, etc. Attainable Is likely to be achieved Results-oriented Generates meaningful, valued results Timed Has an acceptable target date SMART

7. Going further: attainability, moveabilty Short-term outcomes are more likely to show movement… …but consider downstream (medium-term) outcomes if short-term outcomes are not feasibly measured. Outputs Short-term outcomes --------------- Long-term outcomes

8. Indicators What does the outcome look like? How would we know it improved? • Create operational definitions. • Specifics of how the outcome is measured • Make it a number. Quantitative data can be plotted over time or compared across groups. • Rate, percentage • Numerator and denominator • Score (continuous) • e.g., 0 to 100 points • Rating scale (ordinal) • e.g., [0 - ‘never’] -- [1 - ‘sometimes’] -- [2 - ‘always’] Simple counts are not very informative on their own. Find a way to use them as the numerator of a rate or percentage.

9. Timing and duration When will improvement be detected? • Considerations • How long should it take to observe an effect? • What should the effect look like? • Abrupt, sustained improvement • Rises and falls, with gradual trend towards improvement • IC-4: ≥4 quarters of data within 18 months of community engagement • Ensure that the measurement period includes pre-intervention baseline data. • Measure frequently • The more data points, the better. Monthly indicators lend themselves to run/control charts.

10. Level of analysis Where will improvement be seen? • Provider-/initiative-level • A change occurs across several events, people, or organizations. • Process improvements (re: standardization; information transfer) • e.g., inpatient satisfaction with discharge information • Utilization outcomes • e.g, 7-day readmission rates; 30-day primary care follow-up rates • Patient-level • Something happens to an individual person related to his/her patient experience. • Patient activation (e,g., change in PAM score) • Utilization(e.g. , prevented readmission) • Patient-level data should be aggregated to provider-/initiative-level for reporting.

11. Collecting data • Considerations • What method? • Sampling • Survey, case review, etc. • Who collects the data? Who analyzes it? • How will data quality be ensured? • Make it explicit. • Standardization, data specifications • Accountability • Don’t forget to track implementation (process) measures, as well. • Counts, rates among eligible population (offered, refused, completed)

12. More to come • Detecting and reporting improvement • Context and reasons for success/failure

13. Resources • Toolkit – measurement http://www.cfmc.org/caretransitions/toolkit_measure.htm • Measuring Program Outcomes: A Practical Approach http://www.unitedwaystore.com/product/measuring_program_outcomes_a_practical_approach/program_film Excerpts: http://www.unitedwayslo.org/ComImpacFund/10/Excerpts_Outcomes.pdf • ICPCA NCC contact: Tom Ventura tventura@coqio.sdps.org 303-784-5766

14. Questions? CO-ICPCTechnical@coqio.sdps.org The ICPC National Coordinating Center – www.cfmc.org/caretransitions Change Starts Here.