An Algorithm for Testing Unidimensionality and Clustering Items in Rasch Measurement

1. An Algorithm for Testing Unidimensionality and Clustering Items in Rasch Measurement Rudolf Debelak & Martin Arendasy

2. Outline • Aims of this study • PCA and Parallel analysis based on tetrachoric correlations • The proposed algorithm • Procedures • Statistical test • Simulation study • Empirical study • Discussion

3. Aims of Study • Clustering items: exploratory approach to identify items scales with strict criterion • Testing unidimensionality: confirmatory approach to test a unidimensional item set whether yielding a single cluster

4. Literature Review • Commonly used procedures to test unidimensionality: PCA and EFA • Applying to binary data -> based on tetrachoric correlations • Correct number of components/factors -> parallel analysis • Cluster analysis

5. Cluster analysis (from Wikipedia) • Cluster analysis or clustering is the task of assigning a set of objects into groups (called clusters) so that the objects in the same cluster are more similar (in some sense or another) to each other than to those in other clusters. • Hierarchical Cluster Analysis is based on the core idea of objects being more related to nearby objects than to objects farther away. • Measure of similarity (distance)

6. The Basic Structure of the Procedure On Test Item Triplets O3 NOT Maximum Function f Maximum A3 An+1 NOT Maximum Expand Item Set Function f Maximum p less than 0.5 On+1 Ok

7. Assessing the Model Fit • “a function f” is a global fit statistics in this study • The test can be used to evaluate whether the set of items, as a whole, fits the model. ( Suarez-Falcon & Glas, 2003) • First-order statistics (R1): violation of the property of monotone increasing and parallel item characteristic curves • Second-order statistics (R2): violation of the assumptions of unidimensionality and local independence

8. R1C Statistics R1C can be regarded as being asymptotically chi-square distribution with (k – 1)(k – 2) degrees of freedom.

9. Simulation Study • Aim: whether able to detect and reconstruct subsets of items that fit the Rasch model. • Two subsets of items which fit the Rasch model • Six variables were manipulated (next slide) • 10,000 replications were carried out with eRm package which employed the CML estimation method.

10. Variables • The distribution of the item parameters (normal, uniform) • The standard deviations of the item and person parameters • The size of the person sample (250, 500, 1000) • The size of the item set (10, 30, 50) • The correlation between the person parameters (0.0, 0.5)

11. Data Analysis • The proposed algorithm • The PCA based on tetrachoric correlations with parallel analysis (95th percentile eigenvalues)

12. Results (Proposed Method)

13. Results (PCA with Parallel Analysis)

14. Sample Size in PCA • Sample size small than 250 (test length = 10 items) would result in large numbers of indefinite matrices of tetrachoric correlations, making the application of PCA impossible. (Parry & McArdle, 1991; Weng & Cheng, 2005)

15. Empirical Study • The Basic Intelligence Functions (IBF; Blum et al., 2005) • Subtests and Items: verbal intelligence functions (2; 12+15), numerical intelligence functions (2), long-term memory (1; 15), visualization (1; 13). • Between 281 and 284 persons

16. Data Analysis • Using Raschcon for scale-construct with the proposed algorithm. • Andersen likelihood ratios (eRm), fit statistics and PCA on residuals (Winsteps) were calculated. • PCA and parallel analysis of tetrachoric correlations were performed. • all subtests were analyzed separately

17. Results • Proposed algorithm: all subtests identical to the respective subtest; fit to Rasch model. • Andersen tests: fit at .01 level; 3 out of 4 subtests unfit at .05 level. • Mean square in/outfit: all ranged [1.33, 0.65] • PCA on residuals: long-term memory(2.0), others (<1.4) • PCA: long-term memory (2 components)

18. Discussion • A new algorithm was presented and compared with another method PCA of tetrachoric correlations. • R1c statistics: when sample size is large and correlation between latent traits was low, and better in small scales, large variances of item and person parameters. • More preferable than PCA of tetrachoric when sample size is small and scales are large.

19. Further Studies • Systematic comparison with PCA of tetrachoric • Involve more tests for model assumptions • Compare test statistics for the fit of Rasch model • Conduct other IRT models • For the conditions of higher correlation and small sample size, is it possible to find a cut-point (correction) to improve the use of this method?