A New Stopping Rule for Computerized Adaptive Testing

1. A New Stopping Rule for Computerized Adaptive Testing

2. Objective • The predicted standard error reduction (PSER) stopping rule uses the predictive posterior variance to determine the reduction in standard error that would result from the administration of additional items. • The performance of the PSER was compared with that of the minimum standard error stopping rule and a modified version of the minimum information stopping rule. • Results indicate that the PSER makes efficient use of CAT item pools, administering fewer items when predictive gains in information are small and increasing measurement precision when information is abundant.

3. Theoretical Framework • Fixed length stopping rule • Simplicity • Limit the efficiency on unnecessary administration of items • Variable length stopping rules • The standard error (SE) stopping rule • Equivalent measurement precision across the examinee trait continuum • Limit measurement precision even though informative items are still available • The minimum information stopping rule • Prevent the administration of items that contribute little information • Deliver less accurate measurement precision than the minimum SE stopping rule

4. Theoretical Framework • The difficult balance between measurement precision and testing efficiency can subject existing stopping rules to two potential problems that are interest in the current study. • Meaningful gains in measurement precision may be both desirable and possible with the administration of only few additional items. (fixed length and the minimum SE) • Items are unnecessarily administered to examinees for more precise trait estimates are unlikely. • Poor match between pool information and trait distribution (fixed length and the minimum SE) • Good match between pool information and trait distribution (the minimum information)

5. Theoretical Framework

6. Theoretical Framework

7. Theoretical Framework • There is strong motivation to reduce testing burden and high measurement precision for examinees whose trait levels fall within the targeted range on medical and psychiatric CAT assessments. • The new stopping rule, called the predicted standard error reduction (PSER), seeks to balance the dual concerns of measurement precision and testing efficiency by considering the predicted change in measurement precision that would result from the administration of additional items. • The purpose of the research is to explore the properties of the PSER and assess its performance with regard to the number of items administered and the accuracy of theta estimation in comparison with existing stopping rules.

8. Method • The PSER • Hyper(∆SE>= 0.03 but SE is met, continue) • Hypo (∆ SE<0.01 but SE is not met, stop) • The minimum SE stopping rule • The modified minimum information stopping rule (with the minimum SE)

9. Method • Item pool: • mismatched: +1.5 right shifted • matched • 30 and 90 items • 1000 examinees draw from N(0, 1) • GRM • MEPV for item selection and EAP for trait estimation • Criteria • The PSER (0.03 for hyper and 0.01 for hypo)hyper + hypo • The min SE (0.3)base line • The min information (0.5)hypo

10. Results

11. Results

12. Results

13. Results

14. Results

15. Results

16. Conclusion • The new method (PSER) based on the predictive posterior variance. The statistic is computed as part of most fully Bayesian item selection procedures, including the minimum expected posterior variance criterion (MEPV). • The PSER tried to ask one basic question: is it worth continuing? The “worth” is conceptualized as the amount of reduction in uncertainty and measured against two threshold values, the hyper(H+) and hypo (H-) thresholds. • The PSER stopping rule seeks to balance the dual concerns of measurement precision (H+) and testing efficiency (H-) by considering the predicted change in measurement precision that would result from the administration of additional items.