Quality Measures for Rehabilitation: Policy, Provider and Patient Perspectives

1. ACRM-ASNR Pre-Conference Institute Quality Measures for Rehabilitation: Policy, Provider and Patient Perspectives Measuring Clinical Change: Quality Indicators Kenneth J. Ottenbacher University of Texas Medical Branch

2. Disclosures National Institutes of Health Grants: R24 HD065702; R01 AG017638; K12 HD055929; T32 HD007539 PI: Kenneth J. Ottenbacher National Center for Medical Rehabilitation Research (NICHD), National Institute on Neurological Disorders and Stroke, and National Institute on Aging Department of Education Grants: H133G080163; H133P110012 PI: Kenneth J. Ottenbacher National Institute on Disability and Rehabilitation Research

3. Learning Objectives Describe three approaches to measuring clinical change Discuss the difference between clinical change at the individual level and facility level Identify challenges in measuring clinical change

4. Measuring clinically important change is complex. Multiple methods are available with various advantages and disadvantages. How NOT to measure clinically important change: p<.05

5. Statistical Significance versus Clinical Importance Design sensitivity: statistical power for experimental research Mark W. Lipsey The Cult of Statistical SignificanceStephen T. Ziliak & Deirdre N. McCloskey

6. Investigators and clinicians have a long history of interpreting statistically significant findings as also being clinically significant. Example: Smith reports a rehabilitation outcome study with statistically significant results. Jones conducts a replication study with (null) non-significant results.

7. Assume that both trials are well-designed and their internal validity is good. Results: Smith, t = 2.21 (df = 78, p < 0.05) Jones, t = 1.06 (df = 18, p = ns)

8. The interpretation is that Smith found a statistically significant difference (change) in the outcome measure and Jones did not find a statistically significant difference (change). And… that the treatment produced a significant change in the Smith study, but not the Jones study.

9. An alternative way to look at the results of the Smith and Jones studies is to examine the degree to which the null hypothesis is false in both investigations. Cohen (1988) refers to this as the “effect size.”

10. If we compute an effect size for each study, we will find the following: Formula: d = 2(t)/  df Smith d = 0.50 Jones d = 0.50

11. The effect size is the same for the two trials - the degree to which the null hypothesis is false is identical for both the Smith and Jones study. The amount of change associated with the outcome is the same in both studies. Why are the findings in one study statistically significant and the other study non-significant?

12. Things are not always what they appear to be. Power for the Jones trial = 0.18 The chance of a Type 2 error is 0.82 (82%) From: Day, R.H. (1972). Science, Vol. 175, p.1335.

13. Reading made Don Quixote a gentleman, but believing what he read made him insane. G. B. Shaw

14. Measuring Clinical Change • Who is the Target: • Assessment of individual change • Assessment of facility change • Assessment of population change

15. Approaches to Assessing (individual) Change Comparison of change in raw scores or ratings Effect size (standardized mean difference) Reliable change index (Minimal detectable change) Minimal clinically important difference Standardized regression-based change scores

16. There are two approaches to measuring (individual) change. • Distribution-based approaches that rely on expressing change scores in terms of an underlying sampling distribution. • Anchor-based approaches that require an external, independent standard to ‘anchor’ the meaning of clinical importance.

17. Examples: Reliability Change Index (RCI) Need the following information: a) baseline score, b) follow-up score, and c) the standard error of measurement (SEM) for the test. (SEM = S11-r) where S1 = standard deviation, r = reliability coefficient. RCI = (X2 – X1) / SEM The RCI provides an estimate of whether a change is greater than would be expected by chance (measurement error).

18. Examples: Minimal Clinically Important Difference (MCID) Requires identifying a clinically important external standard. For example, 5 motor FIM points. 5 FIM points represents ~ 5% of total scale range on the motor domain. This standard can be based on clinical experience, research literature, patient goals, or some combination. Use of UDS burden of care values to determine external anchor.

19. Mean Decrease in Minutes of Help Per Day per FIMTM Point Change 6 5 4 3 Minutes of Help 2 1 0 13 - 26 26 - 39 39 - 52 52 - 65 65 - 78 78 - 91 MotorFIMScore Ranges Minutes per FIM Point UDSMR Spinal Cord Injury Study, 1990-91 Hamilton et. al.

20. Converting individual measures of clinical change to facility or population measures of clinical change (quality indicators). Examples: Percent of persons in a Case Mix Group achieving Minimal Detectable Change or Reliability Change Index criteria. Proportion of persons in a Case Mix Group achieving a Minimal Clinically Important Difference.

21. Ea M e Gr Ps Ba Si DU Ex DL Co To St Bl WC Bo TuT Ch T ToT Examples (continued): Percent of persons in a Case Mix Group achieving above benchmark expectations. Admission Polar Graph Discharge Benchmark UDSMR

22. Challenges: 1. How to risk Adjust? 2. How to make the process patient centered? 3. What are the unintended consequences?

23. Doing outcomes research is a lot like raising children… you always think you are going to do a better job next time. The End http://rehabsciences.utmb.edu/R24/ACRM.asp