A Hierarchical Framework for Modeling Speed and Accuracy on Test Items

1. A Hierarchical Framework for Modeling Speed and Accuracy on Test Items Van Der Linden

2. Outline • Introduction of speed-accuracy tradeoff • Current Models • Discussion of Models • General Hierarchical Framework • Priors Distributions • Empirical Example • Discussion

3. Introduction of speed-accuracy tradeoff • working faster with lower accuracy or more slowly with higher accuracy

4. Current models • M1: • M2: • M3: • M4: Weibull distribution • M5:

5. Discussion • (1) the within-person level, at which the value of the person parameters is allowed to change over time • (2) the fixed-person level, at which the parameters remain constant; • (3) the level of a population of fixed persons, for which we have a distribution of parameter values across persons • M1 & M2: Confound the within-person and the fixed-person level.

6. (4) speed parameters and time parameters should be separated & incorporated • M2: directly equate speed with the response time • M3 & M5: contain time parameters • M1, M2 & M4: no time parameters

7. General Hierarchical Framework • Key assumptions • (a) a test taker operates at a fixed level of speed & accuracy; • (b) for a fixed test taker, the response and the time on an item are random variables; • (c) separate item and person parameters for the distribution of the response on the items and the time • (d) given the person’s ability and speed, the responses and time are conditional independent • (e) model the relations between speed and accuracy for a population of test takers separately from the impact of these parameters on the responses and times of the individual test takers

8. Levels of Modeling • First-level models: a response model and a response-time model (3PNO) (lognormal) • Joint distribution (locally dependent given a person’s ability and speed)

9. Second-level model: • For person parameters ( ), it was assumed that the and are multivariate normal distributed.

10. Likewise, for item parameters ( ), it was assumed that , , , , and are multivariate normal distributed

11. For the full model, the sampling distribution becomes

12. Identifiablity • Alternative models response model: use IRT models response time model: exponential model, Weibull distribution, constrain some parameters for the second model

13. Priors Distribution • Posterion distribution

14. Empirical Example • 1104 test takers on 96 items in the computerized CPA examination. The items were shown to have a good fit to the 3PL logistic model and the response times to the lognormal model.

15. Results: • , indicating the more able test takers tended to work faster (tradeoff between speed and accuracy)

16. Discussion • Response time • (1) improve testing routines that are traditionally based on the response only. • (2) improve item selection in CAT; • (3) diagnose differential speededness and thus to adjust the problem. • (4) detect aberrant behavior • (5) The hierarchical framework can be used to equate the result from different experiments or results from different conditions of speed.

17. Questions & Future Studies • Scale problem for response model and response time model • Use different models for response model (e.g., 3PL logistic model) and response time model • Consider situations of cheating when response time is rather short…