1 / 24

Detecting Test Security Problems Using Item Response Times and Patterns

Detecting Test Security Problems Using Item Response Times and Patterns. David F. Foster and Dennis Maynes Caveon. Security Problems. Piracy: Stealing questions without concern for score Cheating: Using any method possible to receive an undeserved high score.

piera
Download Presentation

Detecting Test Security Problems Using Item Response Times and Patterns

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Detecting Test Security Problems Using Item Response Times and Patterns David F. Foster and Dennis Maynes Caveon

  2. Security Problems • Piracy: Stealing questions without concern for score • Cheating: Using any method possible to receive an undeserved high score

  3. Cheating and Piracy Can Be Detected • Aberrance statistics indicate the degree to which a test result does not conform to a typical distribution of test results • Measures should include: • Patterns of responding • Item latencies or response times • Similarity to other test results • Test retake patterns and gains/losses • Depends on examinee ability

  4. What you will see: • response pattern aberrance across several ability levels • item latency aberrance across several ability levels • how this kind of information can help solve common security problems

  5. Very Low Theta

  6. Low Theta

  7. High Theta

  8. Very High Theta

  9. What about the situation where the highest probability response is always chosen? Can't see a difference!

  10. Question • How can you tell if a person cheated who achieved 100% on a test? • Answer: Look at item latencies, which works also at all examinee ability levels

  11. Item Latency Analysis • Less affected by examinee ability • Indications of aberrance • Uncorrelated with mean item latencies • Too fast or too slow • Spikes or zero latency across multiple items

  12. Normal Latency Pattern for Low Theta Examinee .3257 = Tau

  13. Aberrant Latency Pattern for Low Theta Examinee -.0578 = Tau

  14. Another Aberrant Latency Pattern for Low Theta Examinee -.0167= Tau

  15. Normal Latency Pattern for Moderate Theta Examinee .5264= Tau

  16. Aberrant Latency Pattern for Moderate Theta Examinee -.0468= Tau

  17. Normal Latency Pattern for High Theta Examinee .3537= Tau

  18. Aberrant Latency Pattern for High Theta Examinee -.0425: Tau

  19. Use of Aberrance Statistics • Help solve testing problems • Help solve test security problems: • Where are the problems occurring? • When did the cheating occur? • Who stole the questions? • How were the questions stolen? …and many others

  20. Let’s Take an Example: Where is Cheating or Piracy Occurring? • General geographic areas • Specific testing sites

  21. General Geographic Areas

  22. Specific Testing Sites

  23. With Information, What Can You Do? • You can: • Investigate further (secret shopping; more data analysis) • Shut down sites • Implement preventative measures • Replace tests • Target areas for anti-piracy education • Take legal action

  24. Conclusions • Your test results maycontain evidence of cheating and piracy. • Analyze your test results often (continuously?) to understand better the nature and scope of any security problems. • Computerized tests bring many advantages and some disadvantages for security. Better aberrance methods will encourage those moving from paper and pencil to computerized testing.

More Related