Assessment of Reliable Change: Methods and Assumptions

Download Presentation

Assessment of Reliable Change: Methods and Assumptions

Loading in 2 Seconds...

- 182 Views
- Uploaded on
- Presentation posted in: General

Assessment of Reliable Change: Methods and Assumptions

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Michael Basso, Ph.D.

Associate Professor and Director of Clinical Training

Department of Psychology—University of Tulsa

Clinical Associate Professor

Department of Psychiatry—University of Oklahoma

- Provide background concerning methods of assessing reliable change
- Describe assumptions and applications of reliable change scores
- Illustrate use of reliable change scores

- Two Basic Approaches
- Assessment of Group Differences Across Time
- Assessment of Individual Differences Across Time

- Assessment of statistically reliable change
- “Does the treatment yield significant benefits for groups of patients?”
- i.e., do average scores at T1 and T2 come from different distributions

- This approach describes the average rate of change over groups primarily
- It is accomplished with repeated measure ANOVA
- Problem: You could have a statistically significant difference with a very small effect size, but it might not be a clinically meaningful change

- “Does the treatment yield significant benefits for groups of patients?”

- Assessment of Clinically Meaningful Change
- “Did the patient’s change in performance at T2 exceed base rates of change?”
- i.e., did the individual show change that exceeded expectations based on measurement error, practice effects, and regression to the mean?

- This method describes the base rate of change
- Change that is exceeds the base rate is not normal, and is therefore clinically meaningful

- Our focus is on the assessment of clinically meaningful change in individuals, but this method can be applied to group data as well

- “Did the patient’s change in performance at T2 exceed base rates of change?”

- How do you establish the base-rate of change?
- Bear into consideration that:
- It would be improbable to obtain the exact same score twice
- There is no perfect test-retest correspondence because of
- measurement error
- regression to the mean
- practice effects

- Reliable change Index scores
- “Does change exceed what would be expected based on measurement error alone?”
- This method is based on Reliability of measurement
- It is used for typical performance tests
- i.e., attitude, personality, psychopathology, etc.

- Standardized Regression-Based Change Scores
- “Does change in scores exceed expectations based on T1 (baseline) scores?”
- This method is based on a validity coefficient (i.e., what T2 score is predicted by the T1 score)
- It is used for maximal performance tests
- i.e., IQ, neuropsychological, etc.

- Elaborated by Jacobson and Truax (1991)
- Based on the standard error of the difference
- Which in turn is based on the reliability coefficient

- This reflects the sampling distribution of difference scores
- it implies the magnitude of differences between two test scores that vary by chance alone

- Based on the standard error of the difference
- Assumptions
- Error components are mutually independent and independent of true pretest and posttest scores
- Error is normally distributed with a mean of 0
- SE of error is equal for all subjects
- These assumptions are questionable in clinical settings (cf. Maassen, 2004)

- To use the RCI, you must compute the SE of difference between two scores
- SEdiff=(2(SD(1-rxx)1/2)2)1/2
- Then, compute a confidence interval for change scores
- for 95% confidence, you multiply 1.96 * SEdiff
- for 90% confidence, you multiply 1.60 * Sediff
- Does the raw score change between T2 and T1 exceed the confidence interval?
- If so, it represents change that exceeds the base rate expected based on measurement error
- Thus, clinically meaningful change has occurred

- If not, then the change is consistent with the base rate expected based on measurement error
- Thus, no clinically meaningful change has occurred

- If so, it represents change that exceeds the base rate expected based on measurement error

- Ferguson, Robinson, & Splaine (2002)
- SF-36 in 200 patients who had undergone a Coronary Artery Bypass Grafting (CABG) surgery
- SF-36 contains 8 scales
- Physical Functioning
- Role Functioning Physical
- Bodily Pain
- General Health
- Vitality
- Social Functioning
- Role Functioning-Emotional
- Mental Health

- Ferguson, Robinson, & Splaine (2002)
- Physical Functioning
- Reliability=.93 (from normative sample of 2474)
- Mean of normative sample=84.15
- SD of normative sample=23.28
- SEdiff=(2(SD(1-rxx)1/2)2)1/2
- SEdiff=(2(23.28(1-.93)1/2 ) 2))1/2=9.85
- 95% CI: (SEdiff)*1.96=19.32
- T1 Mean=40.97
- T2 Mean=63.39
- Mean Diff=22.42
- The mean difference exceeds 19.32
- Thus, clinically meaningful change has occurred as a result of surgery

- Physical Functioning

- Ferguson, Robinson, & Splaine (2002)
- Mental Health
- Reliability=..84 (from normative sample of 2474)
- Mean of normative sample=75.01
- SD of normative sample=21.40
- SEdiff=(2(SD(1-rxx)1/2)2)1/2
- SEdiff=(2(21.40(1-..84)1/2 ) 2))1/2=10.92
- 95% CI: (SEdiff)*1.96=21.40
- T1 Mean=72.08
- T2 Mean=71.84
- Mean Diff=-0.24
- The mean difference fails to exceed 21.40
- Thus, no clinically meaningful change has occurred as a result of surgery

- Mental Health

- Elaborated by Charter (1996)
- Used primarily for maximal performance tests
- The RCI of Jacobsen and Truax is used for typical performance tests
- It assumes that errors between test scores at baseline and time 2 are uncorrelated
- This assumption is untenable in maximal performance tests because of practice effects

- Based on the standard error of prediction
- SEpred=SDY2((1-rY1Y22)1/2)

- The SE reflects the sampling distribution of predicted scores
- It implies the range of scores that might be expected at time two that may be expected from the baseline score and prediction error
- This method requires you to compute the estimated true score
- Y2True=M+((rY1Y2)(Y1-M))

- The T2 score is prone to error, and this formula permits an unbiased estimate of the true score
- The SEpred is used to compute a confidence interval around the estimated true score

- To use the SRB, you must compute the estimated true T2 score
- Compute the confidence interval around this estimated true T2 score
- For 95% confidence, you multiply 1.96 * SEpred
- For 90% confidence, you multiply 1.60 * SEpred
- Does the obtained T2 score fall outside the confidence interval around the estimated true score for T2?
- If so, it represents change that exceeds the base rate expected based on measurement error, regression to the mean, and practice
- Thus, clinically meaningful change has occurred

- If not, then the change is consistent with the base rate expected based on measurement error, practice, and regression to the mean
- Thus, no clinically meaningful change has occurred

- If so, it represents change that exceeds the base rate expected based on measurement error, regression to the mean, and practice

- Basso, Carona, Lowery, & Axelrod (2002)
- WAIS-III re-tested in a group of control subjects over a 3-6 month interval
- FSIQ
- Test-Retest Reliability=.90
- T1 Mean T1=109.4 (11.6)
- T2 Mean T2=115.0 (12.1)
- SEpred=SDY2((1-rY1Y22)1/2)
- SEpred=(12.1((1-.902) 1/2))=5.29
- 95% CI: (SEdiff)*1.96=10.36
- Mean Diff=5.60
- The mean difference fails to exceed the 95% CI
- No individual had a score exceeding the 95% CI

- To apply the SRB, the T2 True Score is estimated
- If the obtained score falls within the CI around the T2 True score, then no clinically meaningful change has occurred

- Basso, Carona, Lowery, & Axelrod (2002)
- An example application:
- T1 obtained score=104
- T2 obtained score=116
- Estimated True T2 Score
- YTrue=M+((rY1Y2)(Y1-M))
- YTrue=100+(.90)(104-100)=103.6
- 116 exceeds 10.36 points from 103.6
- Thus, meaningful change has occurred

- Basso, Carona, Lowery, & Axelrod (2002)
- An example application:
- T1 obtained score=103
- T2 obtained score=106
- Estimated True T2 Score
- YTrue=M+((rY1Y2)(Y1-M))
- YTrue=100+(.90)(106-100)=105.4
- 105 falls within 10.36 points of 106
- Thus, no meaningful change has occurred