1 / 23

Assessment of Reliable Change: Methods and Assumptions

Assessment of Reliable Change: Methods and Assumptions. Michael Basso, Ph.D. Associate Professor and Director of Clinical Training Department of Psychology—University of Tulsa Clinical Associate Professor Department of Psychiatry—University of Oklahoma. Objectives.

libitha
Download Presentation

Assessment of Reliable Change: Methods and Assumptions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessment of Reliable Change: Methods and Assumptions Michael Basso, Ph.D. Associate Professor and Director of Clinical Training Department of Psychology—University of Tulsa Clinical Associate Professor Department of Psychiatry—University of Oklahoma

  2. Objectives • Provide background concerning methods of assessing reliable change • Describe assumptions and applications of reliable change scores • Illustrate use of reliable change scores

  3. Assessment of Clinical Change • Two Basic Approaches • Assessment of Group Differences Across Time • Assessment of Individual Differences Across Time

  4. Assessment of Group Differences Across Time • Assessment of statistically reliable change • “Does the treatment yield significant benefits for groups of patients?” • i.e., do average scores at T1 and T2 come from different distributions • This approach describes the average rate of change over groups primarily • It is accomplished with repeated measure ANOVA • Problem: You could have a statistically significant difference with a very small effect size, but it might not be a clinically meaningful change

  5. Assessment of Individual Differences Across Time • Assessment of Clinically Meaningful Change • “Did the patient’s change in performance at T2 exceed base rates of change?” • i.e., did the individual show change that exceeded expectations based on measurement error, practice effects, and regression to the mean? • This method describes the base rate of change • Change that is exceeds the base rate is not normal, and is therefore clinically meaningful • Our focus is on the assessment of clinically meaningful change in individuals, but this method can be applied to group data as well

  6. Assessment of Clinical Change for Individuals • How do you establish the base-rate of change? • Bear into consideration that: • It would be improbable to obtain the exact same score twice • There is no perfect test-retest correspondence because of • measurement error • regression to the mean • practice effects

  7. Two Methods of Assessing Base Rates of Change • Reliable change Index scores • “Does change exceed what would be expected based on measurement error alone?” • This method is based on Reliability of measurement • It is used for typical performance tests • i.e., attitude, personality, psychopathology, etc. • Standardized Regression-Based Change Scores • “Does change in scores exceed expectations based on T1 (baseline) scores?” • This method is based on a validity coefficient (i.e., what T2 score is predicted by the T1 score) • It is used for maximal performance tests • i.e., IQ, neuropsychological, etc.

  8. Reliable Change Index Scores • Elaborated by Jacobson and Truax (1991) • Based on the standard error of the difference • Which in turn is based on the reliability coefficient • This reflects the sampling distribution of difference scores • it implies the magnitude of differences between two test scores that vary by chance alone • Assumptions • Error components are mutually independent and independent of true pretest and posttest scores • Error is normally distributed with a mean of 0 • SE of error is equal for all subjects • These assumptions are questionable in clinical settings (cf. Maassen, 2004)

  9. Standard Normal Curve—Distribution of Difference Scores

  10. Reliable Change Index Scores--Method • To use the RCI, you must compute the SE of difference between two scores • SEdiff=(2(SD(1-rxx)1/2)2)1/2 • Then, compute a confidence interval for change scores • for 95% confidence, you multiply 1.96 * SEdiff • for 90% confidence, you multiply 1.60 * Sediff • Does the raw score change between T2 and T1 exceed the confidence interval? • If so, it represents change that exceeds the base rate expected based on measurement error • Thus, clinically meaningful change has occurred • If not, then the change is consistent with the base rate expected based on measurement error • Thus, no clinically meaningful change has occurred

  11. Reliable Change Index Scores—An Example • Ferguson, Robinson, & Splaine (2002) • SF-36 in 200 patients who had undergone a Coronary Artery Bypass Grafting (CABG) surgery • SF-36 contains 8 scales • Physical Functioning • Role Functioning Physical • Bodily Pain • General Health • Vitality • Social Functioning • Role Functioning-Emotional • Mental Health

  12. Reliable Change Index Scores—An Example • Ferguson, Robinson, & Splaine (2002) • Physical Functioning • Reliability=.93 (from normative sample of 2474) • Mean of normative sample=84.15 • SD of normative sample=23.28 • SEdiff=(2(SD(1-rxx)1/2)2)1/2 • SEdiff=(2(23.28(1-.93)1/2 ) 2))1/2=9.85 • 95% CI: (SEdiff)*1.96=19.32 • T1 Mean=40.97 • T2 Mean=63.39 • Mean Diff=22.42 • The mean difference exceeds 19.32 • Thus, clinically meaningful change has occurred as a result of surgery

  13. Reliable Change Index Scores—An Example • Ferguson, Robinson, & Splaine (2002) • Mental Health • Reliability=..84 (from normative sample of 2474) • Mean of normative sample=75.01 • SD of normative sample=21.40 • SEdiff=(2(SD(1-rxx)1/2)2)1/2 • SEdiff=(2(21.40(1-..84)1/2 ) 2))1/2=10.92 • 95% CI: (SEdiff)*1.96=21.40 • T1 Mean=72.08 • T2 Mean=71.84 • Mean Diff=-0.24 • The mean difference fails to exceed 21.40 • Thus, no clinically meaningful change has occurred as a result of surgery

  14. Standardized Regression Based Change Scores • Elaborated by Charter (1996) • Used primarily for maximal performance tests • The RCI of Jacobsen and Truax is used for typical performance tests • It assumes that errors between test scores at baseline and time 2 are uncorrelated • This assumption is untenable in maximal performance tests because of practice effects

  15. Standardized Regression Based Change Scores • Based on the standard error of prediction • SEpred=SDY2((1-rY1Y22)1/2) • The SE reflects the sampling distribution of predicted scores • It implies the range of scores that might be expected at time two that may be expected from the baseline score and prediction error • This method requires you to compute the estimated true score • Y2True=M+((rY1Y2)(Y1-M)) • The T2 score is prone to error, and this formula permits an unbiased estimate of the true score • The SEpred is used to compute a confidence interval around the estimated true score

  16. Standard Normal Curve—Distribution of Standard Error of Prediction Around Estimated True Score

  17. Standardized Regression Based Change Scores--Method • To use the SRB, you must compute the estimated true T2 score • Compute the confidence interval around this estimated true T2 score • For 95% confidence, you multiply 1.96 * SEpred • For 90% confidence, you multiply 1.60 * SEpred • Does the obtained T2 score fall outside the confidence interval around the estimated true score for T2? • If so, it represents change that exceeds the base rate expected based on measurement error, regression to the mean, and practice • Thus, clinically meaningful change has occurred • If not, then the change is consistent with the base rate expected based on measurement error, practice, and regression to the mean • Thus, no clinically meaningful change has occurred

  18. Standardized Regression Based Change Scores--An Example • Basso, Carona, Lowery, & Axelrod (2002) • WAIS-III re-tested in a group of control subjects over a 3-6 month interval • FSIQ • Test-Retest Reliability=.90 • T1 Mean T1=109.4 (11.6) • T2 Mean T2=115.0 (12.1) • SEpred=SDY2((1-rY1Y22)1/2) • SEpred=(12.1((1-.902) 1/2))=5.29 • 95% CI: (SEdiff)*1.96=10.36 • Mean Diff=5.60 • The mean difference fails to exceed the 95% CI • No individual had a score exceeding the 95% CI • To apply the SRB, the T2 True Score is estimated • If the obtained score falls within the CI around the T2 True score, then no clinically meaningful change has occurred

  19. Standardized Regression Based Change Scores--An Example • Basso, Carona, Lowery, & Axelrod (2002) • An example application: • T1 obtained score=104 • T2 obtained score=116 • Estimated True T2 Score • YTrue=M+((rY1Y2)(Y1-M)) • YTrue=100+(.90)(104-100)=103.6 • 116 exceeds 10.36 points from 103.6 • Thus, meaningful change has occurred

  20. Standardized Regression Based Change Scores--An Example • Basso, Carona, Lowery, & Axelrod (2002) • An example application: • T1 obtained score=103 • T2 obtained score=106 • Estimated True T2 Score • YTrue=M+((rY1Y2)(Y1-M)) • YTrue=100+(.90)(106-100)=105.4 • 105 falls within 10.36 points of 106 • Thus, no meaningful change has occurred

  21. Questions?

More Related