1 / 20

Exploring new versions of DIMTEST for use with Polytomous Data

Exploring new versions of DIMTEST for use with Polytomous Data. Tan Li Louis Roussos Measured Progress July 24, 2009. Outline. Introduction Dimensionality Hypothesis Test Poly-DIMTEST Methods Poly-DIMTEST without AT2 Poly-NEWDIM Simulation Study Results & Conclusions Future Work.

darena
Download Presentation

Exploring new versions of DIMTEST for use with Polytomous Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Exploring new versions of DIMTEST for use with Polytomous Data Tan Li Louis Roussos Measured Progress July 24, 2009

  2. Outline • Introduction • Dimensionality Hypothesis Test • Poly-DIMTEST • Methods • Poly-DIMTEST without AT2 • Poly-NEWDIM • Simulation Study • Results & Conclusions • Future Work

  3. Introduction • Dimensionality Hypothesis Test • Important assumption for many IRT models • Equating • Scoring • Scaling • Calibration • DIF analysis • Hypothesis Test • H0: dE = 1 vs. H1: dE > 1

  4. Introduction • Poly-DIMTEST (Nandakumar, Yu, Li, & Stout, 1998) • Hypothesis test • H0: vs. H1: • Split all the test items into three subtests: AT1, AT2 and PT • The test statistic: • Stand Error of CCOV comes from a complicated formula

  5. Introduction • Poly-DIMTEST • Weaknesses • Difficulty on finding and choosing AT2 items • Not enough items left for PT

  6. Methods • Poly-DIMTEST without AT2 • Based on dichotomous version of DIMTEST without AT2 • (Stout, Froelich, & Gao, 2001) • Steps • Split all the test items into two subtests: AT and PT • Fit a unidimensional nonparamatric model to the original data by kernel smoothing • Simulate N samples from the model to take the place ofAT2 • The test statistic: • Stand Error of CCOV comes from the same formula provided by Nandakumar, et al. (1998)

  7. Methods • Poly-NEWDIM • Based on dichotomous version of NEWDIM(Seo & Roussos, 2009) • Similar procedure with Poly-DIMTEST without AT2 • The test statistic: • Standard Errorcomes from the Standard Deviation over the simulated samples

  8. Simulation Study • Dichotomous items • All of the parameters were randomly generated from the distributions based on • real data from a large multi-year pool of 729 grade 5 math items

  9. Simulation Study • Polytomous items • All of the parameters were randomly generated from the distributions based on • real data from a large multi-year pool of 729 grade 5 math items

  10. Simulation Study • Type I Error Study • Power Study • 2 dimensions simple structure

  11. Simulation Study • Factors • 500 examinees and 1000 examinees • 52 pts test and 32 pts test • AT subtest • 52pts test: 5 MC, 10 MC, 2 CR, and 5 CR items • 32pts test: 3 MC, 6 MC, and 3 CR items

  12. Results

  13. Results Type I Error – 52 pts test, 400 trials 500 Examinees 1000 Examinees < 3 [3,7] >7

  14. Results Type I Error– 32 pts test, 400 trials 500 Examinees 1000 Examinees < 3 [3,7] >7

  15. Results Power– 52 pts test,400 trials 500 Examinees 1000 Examinees < 85 ≥85

  16. Results Power– 32 pts test, 400 trials 500 Examinees 1000 Examinees < 85 ≥85

  17. Conclusion • Type I error study • Conservative Type I error behavior • Poly-NEWDIM performs closer to nominal (0.05). • Power study • Poly-NEWDIM has greater power than Poly-DIMTEST without AT2 • Poly-NEWDIM provides adequate power for a variety of conditions.

  18. Future Work • More examinees • Dimensionality structure • Item parameter simulation models • Develop a method to choose AT subtest for mixed MC and CR tests • Real datasets • Skewed ability distributions

  19. Reference • Nandakumar, R., Yu, F., Li, H., & Stout, W. (1998). Assessing Unidimensionality of Polytomous Data. Applied Psychological Measurement, 22, 99-115. • Stout, W., Froelich, A., & Gao, F. (2001). Using Resampling Methods to Produce an Improved DIMTEST Procedure. Essays on item response theory, 357-375 • Seo, M., & Roussos, L. (2009). Evaluation of DIMTEST Effect-Size Measure and Its Application. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.

  20. Acknowledgement Measured Progress Department of Psychometrics Dr. Louis Roussos

More Related