Reporting item r esponse theory r esults Jeffrey B. Brookings Wittenberg University

1. Reporting item response theory resultsJeffrey B. BrookingsWittenberg University Presented at the SAMR/SWPA Symposium: Handy tips for communicating and reporting your findings April 5, 2013

2. Ph.D. Comics, 2013

3. Item Response Theory • Mathematical models that probabilistically describe the relation between a person’s response to an item and his/her standing on a latent trait. • The Rasch model—a “one-parameter” model (difficulty)—locates person ability and item difficulty on the same scale (logits or log odds). • “…a person having a greater ability than another person should have the greater probability of solving any item of the type in question, and similarly, one item being more difficult than another means that for any one person the probability of solving the second item is the greater one.” (Rasch, 1960, p. 117) • The purpose of Rasch analysis is to produce unidimensional measures that cover a wide range of the latent trait.

4. Reporting results froma Rasch analysis • Item and scale descriptive statistics • PCA of standardized residuals following extraction of the Rasch component (test for unidimensionality) • Item “difficulty” estimates (in logits) • Item fit statistics • Item characteristic curves (ICCs) • Category response curves (CRRs) • Person/item map • Person/item separation reliability

5. The Psychosocial Risk Factor Survey(Eichenauer, Feltz, Wilson, & Brookings, 2010) • Assesses psychosocial risk factors for cardiac disease • 70 items, 5-point response scale: 0 - “Strongly Agree” to 4 - “Strongly Disagree” • Scales: Depression, Anxiety, Hostility, Social Isolation, and Emotional Guardedness. • Analyses: Responses to the 14 Depression Scale items (340 patients from five cardiac rehabilitation programs in the Midwest)

6. Rasch Item Statistics

7. PCA of Standardized Residuals Total raw variance in observations 29.5 100.0% Raw variance explained by measures 15.5 52.6% Raw variance explained by persons 6.4 21.6% Raw variance explained by items 9.2 31.0% Raw unexplained variance (total) 14.0 47.4% Unexplained variance in 1st contrast 1.9 6.4% Unexplained variance in 2nd contrast 1.6 5.5% Unexplained variance in 3rd contrast 1.3 4.4% Unexplained variance in 4th contrast 1.2 4.2% Unexplained variance in 5th contrast 1.2 4.0%

9. Item characteristic curve—with 95% CI—for item 27: “My thoughts feel so scattered lately”

10. Item characteristic curve—with 95% CI—for item 12:“I think more about ending my life lately”

11. Rasch Category Responses

12. Person/Item Map Mean person measure = -0.94 Mean item measure = .00

13. Reliability • Person separation reliability – Analogous to Cronbach’sα; degree to which the scale differentiates persons; range 0 – 1 • For PRFS Depression: .88 • Item separation reliability – Degree to which item difficulties are differentiated; range 0 – 1 • For PRFS Depression: .99

14. Summary of Rasch Analysis forthe PRFS Depression Scale • Good evidence for unidimensionality • Mean point-measure r = .626 • Acceptable person and item separation reliabilities (.88 and .99, respectively) • Some misalignment of persons and items • One mis-fitting item: #12 (“I think more about ending my life lately”)

15. Recommended Reading Bond, T.G., & Fox, C.M. (2007). Appling the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Erlbaum.