“Get Ready for Continuous Computer Based Testing” Your psychometrician can be used as a flotation device

1. “Get Ready for Continuous Computer Based Testing”Your psychometrician can be used as a flotation device Troy Elliott Association of Social Work Boards 2007 Annual Conference

2. Today’s Presentation I. Idle Ruminating II. Rudderless blathering III. Nebulous opining IV. Insipid Generalizing Council on Licensure, Enforcement and Regulation

3. Who We Are Association of Social Work Boards (ASWB) 59 members (49 States, D.C., USVI, 8 Provinces) 4 Examinations for licensure About 27,000 administrations annually Other services (model law, trainings, etc.) Council on Licensure, Enforcement and Regulation

4. About the Tests 170 item (150 scored), 4 option m.c., 4 hrs. Computer based, continuous National passing scores Pre-equated— “unofficial score” available to candidate at test site ACT, Inc.—psychometric and administrative support (200+ test centers) Council on Licensure, Enforcement and Regulation

5. Our Fascinating History, or “Wake Me at the Next Slide” First test, 1983 Paper and pencil until mid ‘90s First non-p&p – portable device Contract with ACT began in 1997 Continuous CBT began 1998 Blah blah blah Etc. Council on Licensure, Enforcement and Regulation

6. What We Had Going for Us • Relatively short test (170 items) • Relatively simple format (m.c.) • National passing scores; IRT already in use • Steady – growing test population • Budget already “exam heavy” (70 – 80%) • Customer demand (social workers and boards) Orson Welles: someone who had a lot going for him. Council on Licensure, Enforcement and Regulation

7. What we had going against us • Item bank work needed • Generous testing times • Customer apprehension • Culture of accommodation, responsiveness • General naivete Council on Licensure, Enforcement and Regulation

8. One additional challenge - CRC ASWB Candidate Registration Center • All exam registrations come to us • We screen for eligibility to take exam (except in 4 states) • Forward data to ACT, candidates • Candidates schedule appt. with ACT Council on Licensure, Enforcement and Regulation

9. Problems and Solutions Item Exposure 3 forms online at all times 3 forms online Pre-equating Pre-equating Aggressive pretesting Aggressive pretesting Strong item production system Council on Licensure, Enforcement and Regulation

10. What it’s All About: ITEMS What it’s NOT all about: The Hokey Pokey Council on Licensure, Enforcement and Regulation

11. Getting Items Lots of ways to do this: our process Our words of advice • Be realistic – account for high item attrition Bottom line • A bunch of items isn’t enough Council on Licensure, Enforcement and Regulation

12. Test Administration: The Seamy Underbelly • TEST SITES • Enough? Locations? Multiple users okay? • ASWB: 27K administrations = 108,000 hrs./year, about 230 test sites • Quality of test sites? Security ? • ASWB: extensive security measures through ACT Technology at test sites? Score reporting? ASWB: tests downloaded at appt. time; “unofficial scores”, including diagnostics, at conclusion of test Council on Licensure, Enforcement and Regulation

13. Test Administration: The Seamy Underbelly The Front End – Registration Who’s going to do it? You? Your contractor? Someone else? PROS and CONS Council on Licensure, Enforcement and Regulation

14. Test Administration: The Seamy Underbelly • Ongoing Issues • Continuous testing and the ADA • Continuous testing and ESL • Problems along the way: cancellations, changes, refunds • Problems at the test site • Retakes and waiting periods Council on Licensure, Enforcement and Regulation

15. Test Administration: The Seamy Underbelly • After the test • Score reporting to boards • “Unofficial” vs. “Official” • Failing score reports • Complaints and appeals Council on Licensure, Enforcement and Regulation

16. Hindsight The Good Service to members, service to candidates, steady revenue stream, greater understanding/familiarity with your organization over time The Bad Security risks, infrastructure demands, item demands, costs, dependence on vendors The Ugly Managing expectations, anticipating problems, searching for loopholes Council on Licensure, Enforcement and Regulation

17. Hindsight • Items—now and ongoing • Equating • National passing score • Registration/scheduling • system • Adequate test sites, security • Enforceable policies on retakes, appeals, ADA, ESL, technical problems, etc. Are YOU ready to take the plunge? Council on Licensure, Enforcement and Regulation

18. Speaker Contact Information For more information Troy Elliott Communications Director ASWB telliott@aswb.org 1-800-225-6880, ext. 3005 Council on Licensure, Enforcement and Regulation