1 / 25

Efficient Test Assembly for Large-Scale Assessment: A Hybrid Approach

This method combines Automated Test Assembly (ATA) with content experts' input to create optimal test forms meeting psychometric targets and content constraints. Genetic Algorithm optimizes item selection. Improve feedback and reduce distance from target TIF and TCC. Useful for ELA and Math tests.

tala
Download Presentation

Efficient Test Assembly for Large-Scale Assessment: A Hybrid Approach

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Method for Cooperative Test Assembly for Large-Scale Assessment Jon Brasfield Wonsuk Kim Matt Finkelman Louis Roussos

  2. Automated Test Assembly (ATA) • Foundation: Birnbaum (1968) suggested using the additive relationship between Item Information and the TIF to assemble a test • General outline, not actual method • Computers weren’t powerful enough

  3. ATA Literature • Lots of ATA literature • Doesn’t involve content experts • Need for a method that combines the benefits of ATA with the substantive contributions of content experts • More practical

  4. Outline of Project • Use ATA to improve the feedback loop inherent in Form Pulling • Will automatically create forms that meet psychometric targets (TCC and TIF) and content constraints • ELA Test – Create form options • Math Test – Create initial form

  5. ELA Test • Create detailed feedback to give to test developers • Based on initial form – constructed primarily on content concerns • Developers create initial form, denote “definite” and “maybe” items • Content Constraints: item type, passage association, standard addressed • Psychometric constraint: Target TCC tolerance

  6. ELA test – Our process Item Pool 5,000 Times Test Form

  7. ELA Test – Our process Item Pool 5,000 times Test Form

  8. ELA Test – Our Process • Must meet tolerance on Target TCC

  9. Initial Form

  10. Information: Initial form

  11. Results from random replacement

  12. ELA Test – Best Result • Used Genetic Algorithm • Based on theories of evolution and natural selection • Originally discussed in Computer science (Holland, 1968) • Potential solutions “evolve” toward a target, or optimum solution • Require 3 components: Decision variable, objective function, and constraints

  13. ELA Test – best result • Our objective function: minimize total distance from Target TIF • A set of items that meets the constraints is selected (a “parent”) • For each parent, each item is temporarily replaced (one at a time) with a random item from the pool • Process is repeated, selecting the fittest sets each time

  14. Genetic Algorithm – “best” solution

  15. Actual Form

  16. Information: Actual form

  17. Information comparison

  18. Summed absolute distance comparison

  19. Bottom Line: ELA Test • All forms found exhibited a lower summed absolute distance from the target than the initial form or the actual form Target TIF Improvement

  20. Math Test • Initial form provided by psychometrics: Optimize statistics while staying within content constraints • Constraints include Item type, strand, standard, age of item • Only one form is needed, so GA is used

  21. Math Results

  22. Math Results

  23. Math Test – Bottom Line TTIF Distance Reduction of 89.17% TTCC Distance Reduction of 91.68%

  24. Summary • ELA Test – Content Experts can provide an initial form, psychometrics can give multiple statistically sound options while keeping important items • Math Test – Can easily construct a “first draft” form that is very close to psychometric targets and meets content constraints

  25. Future research • Qualitative feedback from content experts, DOE • Use Genetic Algorithm to develop multiple forms

More Related