1 / 21

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA. COMPUTER-ASSISTED TESTING PETYA GEORGIEVA BULGARIAN STANAG 6001 TESTING TEAM HEAD 2 – 6 September 2013 SHAPE (Mons) Belgium. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA. BG CAT Background.

gitano
Download Presentation

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTINGPETYA GEORGIEVABULGARIAN STANAG 6001 TESTING TEAM HEAD2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

  2. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA BG CAT Background till 1 March, 2012 – approbation of the CAT “writing” module; till 1 July, 2012 – analysis of the results; since exam session Autumn 2012 – optional CAT “writing” module; March, 2013 – approbation of CAT modules for “reading” and “listening”; till 1 July, 2013 – analysis of the results; since exam session Autumn 2013 – optional CAT “writing”, “reading”, and “listening” modules; since exam session Autumn 2014 – all candidates are to be tested on computers for the modules “writing”, “reading”, and “listening”(Action Plan 1 iaw Ministerial Order № 857/10.11.2011). since exam session Autumn 2013 – parallel pen-and-paper and CAT in the three skills. As of Spring 2015 - End of pen-and-paper STANAG 6001 tests (Action Plan 2 iaw Ministerial Order № 219/19.03.2013 ). Petya Georgieva

  3. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA • Administrative Module main menus: • Lists • Protocols • Tasks • Tests • Evaluation and Analysis Petya Georgieva

  4. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Petya Georgieva

  5. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Petya Georgieva

  6. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Candidates’ module: • Reading Test • Listening Test • Writing Test Petya Georgieva

  7. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Petya Georgieva

  8. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Petya Georgieva

  9. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA • follows the current Specifications for STANAG 6001 exam for the Bulgarian Armed Forces; • starts with items for L1, then L2, then L3 ; • a minimum of 12 and a maximum of 18 questions per level; • level is given for a minimum of 70% correct answers; • a candidate can terminate the exam when a level is achieved or decide to go up; • 50%-69% correct answers of the next level → (+)level. Computer-assisted test evaluation algorithm Petya Georgieva

  10. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Paper-based withdrawals -16.67% Failed – Reading (1.5%) Listening (2.5%) Writing (1%) CAT withdrawals – 17.16% Failed – Reading (2.56%) Listening (16.21%) Writing (0%) Statistics and comparison between CAT and paper-based tests Petya Georgieva

  11. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Statistics and comparison between CAT and paper-based tests Petya Georgieva

  12. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Statistics and comparison between CAT and paper-based tests Petya Georgieva

  13. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA CAT – 33% of candidates improved their results (R, L, or both). Reading – 30.76% showed lower results; 33.33% confirmed the results; 35.91% scored higher. Listening – 27.92% showed lower results; 39.63% confirmed the results; 32.45% scored higher. Data refers to March 2013 CAT piloting session (R and L). Petya Georgieva

  14. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Petya Georgieva

  15. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Petya Georgieva

  16. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA PROS: • candidates can manage their time; • decreased candidates’ frustration; • time saving (for both candidates and testers); • anonymity criteria unconditionally complied with; • immediate feedback; • highly objective feedback; • fewer items are compromised; • new items can be piloted and statistics is facilitated. Petya Georgieva

  17. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Cons: • interface problems for the test-takers (computer screen clarity, font, size, color of the letters) • due to test-takers’ characteristics (computer literacy, motor skills needed to respond to test items, danger of bad time-management); • no option for test administrators to troubleshoot during the test process. Petya Georgieva

  18. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA RECOMMENDATIONSFUTURE ACTION PLANS (based on candidates retrospective written and oral feedback and test administrators’ impressions); Several groups: • technical (software perfection, interface layout); • methodological (revision of test components and test administration) • facility use and administrative support Petya Georgieva

  19. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA • Instructions – reviewed; • Layout of the reading material – black font; screen allowing for parallel viewing of the text and answer sheet; • Increase time allotted for looking through the options for L1 and L2 candidates at the Listening; • Reducing number of items if more than 3 per text (R and L). Petya Georgieva

  20. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Thank you for your attention! Petya Georgieva

  21. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA ? ? ? ? ? ? Petya Georgieva

More Related