Testing in the time of crisis

1. BILC PROFESSIONAL SEMINAR Stockholm , October 14 - 17, 2013 INNOVATIVE TEST DESIGNS AND FORMATS Testing in the time of crisis Lt.Col. F. Gratton

2. course of action Summary: • Past situation (up to Sept 2013) • New Course Of Actionsadopted • A do-it-all software • Proposals

3. Stanag Proficiency Test 1.0 RECEPTIVE SKILLS (Listening & Reading) 1: Multilevel test (level 1 to 4) 2: Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4: Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correctanswersmultiplied for a coefficient 7: Potentialuse of “F” factor

4. How STANAG levels wereawarded(Stanag Proficiency Test 1.0) RECEPTIVE SKILLS # of correctanswers Multipliedby fixedcoefficient (1,66)

6. PRODUCTIVE SKILLS Proficiency Test 1.0 SPEAKING(holisticallyassessed) 1: Functionallanguageassessed in a global manner 2: Structured interview 3: Tailored to the candidate 4: Checks & probes 5: 1 to 2 roleplays

7. JFLT 1.0 PRODUCTIVE SKILLS WRITING (holisticallyassessed) Three tasks (one for each level)

8. Summary: • Past situation • New COAsadopted • A do-it-all software • Proposals

9. New COAsadopted • Specifications • Cut-off score • Joint Database

11. PURPOSE • LEVELS OF LINGUISTIC KNOWLEDGE • TEST CONTENT • TEST FORMAT • VALIDATION PROCEDURES • ADMINISTRATION PROCEDURES

12. Stanag Proficiency Test 1.0 TWAS HIGH TIME!!! RECEPTIVE SKILLS (Listening & Reading) 1: Multilevel test (level 1 to 4) 2: Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4: Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correctanswersmultiplied by a coefficient 7: Potentialuse of “F” factor

13. Stanag Proficiency Test 2.0 New Key-factors • Eachsection is a mini-test (L & R) • Plus levels • Percentages

14. New COAsadopted • Specifications • Cut-off score • Joint Database

15. JFLT 2.0: RDS (Listening & Reading)

16. EXAMPLE • Level 1 (15 questions) • Level 2 (15 questions) • Level 3 (15 questions) • Max score: 45 Allcandidatesanswercorrectly to 30 questions With Old test theywouldallget the same score

18. New COAsadopted • Specifications • Cut-off score • Joint Database

19. NEW LISTENING & READING ITEMS JDB (JOINT DATA BASE) JOINT EFFORT HOWNOT TOMAKE JOINT EFFORTS

20. Test-writersinvolved • Accustomed to militaryenvironment • Language Testing Seminar • Qualified • Norming Sessions

21. THE JDB FLOW AIR FORCE ARMY NAVY CARABINIERI

22. 2012 2013 SEP OCT NOV DEC JAN FEB MAR APR MAJ JUN JUL AUG WG GdL GdL GdL Phase3 Phase1 Phase2 Phase4 MAJ 2013 40 NEW ITEMS. INTO JDB • PREPARATION OF FIRST BATCH OF ITEMS • SENT TO OTHER GROUP AUGUST 2013 60 NEW ITEMS. INTO JDB FEBRUARY 2013 20 NEW ITEMS. INTO JDB NOVEMBER 2013 80 NEW ITEMS. INTO JDB TIMINGS PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) 2nd Group 1st Group 4th Group 3rd Group FINAL MODIFICATION APPROVAZIONE TESTERS MEET FOR 1° REVISION Phase1 Phase1 Phase2 Phase3 Phase4 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) • REPERIMENTO • PREPARAZIONE # 120 TTIVAZIONI • INVIO AD ALTRO GRUPPO • PREPARATION OF FIRST BATCH OF ITEMS • SENT TO OTHER GROUP FINAL MODIFICATION APPROVAZIONE TESTERS MEET FOR 1° REVISION • PREPARATION OF FIRST BATCH OF ITEMS • SENT TO OTHER GROUP Phase1 Phase2 Phase3 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) TESTERS MEET FOR 1° REVISION

23. Summary: • Past situation • COA (specs, JDB, cut-off score) • A do-it-all software • Proposals

24. WHAT’S THE DIFFERENCE ?

25. PC-assessment-related terminology

26. the software is used for: • Needs analysis:(surveys) • Placement test • Any training activity • Assessment: • First level survey • Post-course • Pre-certification • Certification

27. Software’s system Schema Create questions And organize them in tests using a windows based PC

28. Assessment …… … allows to choose: • Time limits • Feedback to test-taker • Styles (Template) • Jumps • Question shuffling • Instructions to test-takers On Windows PC Questions via Browser Assessment Definitions

29. Assessments su PC Windows • … also created with authoring manager by selecting Qs previously created • Any question can be chosen from the database via Browser Questions Assessment Definitions

30. Software’s system Schema Result reports, CIA, graphs, gimmicks, you name it ….. Create questions And organize them in tests using a windows based PC Set security parameters, schedule assessment and link to other (Learning Management Systems assessment published using any browser, secure browsers, or a PC/MAC

31. Types of Questions: Multiple Choice

32. LikertScale (for questionnaires)

33. Essay Question • Candidate can write free text in the space provided • Testers will evaluate later

35. Summary: • Past situation • COA (specs, JDB, cut-off score) • A do-it-all software • Proposals s ing

36. Testing in the time of crisis • Wide project • Sharingexperience & capabilities • Optimization of resources • No alternative to B.A.T.

38. COMBINED DATABASE or … A multilateral-based CDB or … A Bilateral-based CDB

40. 1 2 SPECS FLOW 3 TIME SCHEDULE

42. canutesting@sclingue.esercito.difesa.it Thank you francesco.gratton@gmail.com