Download
1 / 16
160 likes | 264 Views
Reliability of Individual Standard Deviations. Ryne Estabrook Kevin Grimm Preparation for GSA, Nov 16-20 Design and Data Analysis Nov 2, 2006. Individual Standard Deviations. Measure of intraindividual variability. Scaled in the units of the measure. Easily calculable and interpretable.
E N D
Reliability of Individual Standard Deviations Ryne Estabrook Kevin Grimm Preparation for GSA, Nov 16-20 Design and Data Analysis Nov 2, 2006
Individual Standard Deviations • Measure of intraindividual variability. • Scaled in the units of the measure. • Easily calculable and interpretable. • Measures of intraindividual variability are useful both as outcomes and predictors. • Used in cases where meaningful interoccasion variability exists, but shows no discernable trend.
Problems with ISDs • ISDs are treated as theoretically identical to means, and differences between them are not checked. • Classical test theory violations: • ISDs assume meaningful true score variance. • CTT assumes all variance is error variance. • Means and ISDs may be differentially reliable. • ISDs may require more occasions to be reliable than means do.
Objectives of the Simulation • To explore how true-score variance affects reliability, simulations were carried out. • Discern the requisite number of occasions and reliability of the original test to produce valid ISDs. • Discover the variance conditions under which ISDs are most reliable, and allow researchers to estimate reliability.
Setting Up the Simulation • For each iteration: • A sample of 100 records are assigned both a mean and an ISD from a bivariate normal distribution. • Means are distributed in the population N(0,k). • ISDs are distributed in the population N(f(k), g(k)) such that f(k) and g(k) vary across iterations. • μISD = [0.5-5.0] * σMEAN, σISD = [.25-2.5] * σMEAN • μISD≥3σISD to keep the distribution of ISDs above zero. • Alterations to the ranges of the above parameters did not significantly alter results.
Setting Up the Simulation • For each iteration: • A number of observations (t) and test reliability (ρ) were assigned. • t within [3:103], ρ within [.09:.99] • Each record is observed t times, with an observed score at every occasion calculated from μj and σj. • Observed Scoreij = √ρ* Trueij + √(1-ρ) * Errorij • An observed mean and ISD is calculated for each record. • Each dataset thus included 100 records for each possible combination of μISD, σISD,t and ρ. • The procedure was repeated for 75 datasets.
First Simulation • Investigate the reliability of across occasion means. • Correlation between true and observed means constitutes the reliability index. • Regress reliability of the across occasion mean on the μISD,σISD, number of occasions and ρMEASURE. • Stepwise regression of the parameters, their transformations and 2-way interactions yielded an average R2 of .936.
Results - Mean μISD = .5 * σMEAN μISD = σMEAN μISD = 1.5 * σMEAN
Results - Mean • Inclusion of intraindividual variation: • Increasing μISD negatively affects reliability. • Increasing σISD has a negligble negative affect on reliability. • May result from a non-linear effect of μISD and the correlation between μISD and σISD . • Highlights the CTT issue: • Any increase in variance is treated as error variance, decreasing the estimated reliability.
Results - Mean • Increasing the number of observations provides the greatest benefit to reliability. • Reliability of the original measure has little effect relative to other predictors. • Multiple iterations of identical tests serve to increase the test length by a factor of t. • Test reliability indicates the within occasion maximum for reliable measurement; aggregating increases reliability very quickly.
Second Simulation • Investigate the reliability of across occasion individual standard deviations. • Correlation between true and observed ISDs constitutes the reliability index. • Regress reliability of the ISD on the μISD,σISD, number of occasions and ρMEASURE. • Stepwise regression of the parameters, their transformations and 2-way interactions yielded an average R2 of .955.
μISD = 2 * σISD μISD = 3 * σISD μISD = 4 * σISD μISD≥ 6 * σISD Results - ISD
Results - ISD • Increasing the ratio of σISD to μISD increases reliability. • Increasing σISD and μISD in proportion changes the units, but not the meaning, of ISDs. • The number of occasions has a greater effect for ISDs than for means. • The reliability of the measure used again has a negligible effect.
Conclusions • Reliability of means and ISDs depend on the relationship between μISD,σISD & σMEAN. • Means are most reliable when variance (μISD or error) is minimized. • ISDs are most reliable when σISD is maximized relative to μISD and, to a lesser extent, σMEAN. • ISDs are less reliable than means. • ISDs based on very few occasions may be too unreliable to interpret.
Implications • Tests that are reliable for mean levels may not be reliable for measuring variability. • ISDs from few occasions lack statistical power. • ISDs do not differ theoretically from single-occasion population standard deviations. • Standard deviations based on 10 observations of an individual are as reliable as a standard deviation from a sample of 10.
Thanks • John Nesselroade • Current and former colleagues at CDHRM at the University of Virginia • NIA T32 AG20500-01
