Raters Training as a Safety Concern

1. Associate Member Raters Training as a Safety Concern ICAO LPR Implementation Workshop Russia, St-Petersburg, May 24-26, 2011 Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director

2. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern - RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OFRATER’S ADEQUACY

3. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

4. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Aviation English test results are felt around the globe Raters Training as a Safety Concern

5. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 3 instead of Level 4: Loss of job and/or occupation Financial losses for AO or ANSP Market losses Redistribution of air transportation services Lack of confidence for test-takers Raters Training as a Safety Concern

6. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 4 instead of Level 3: Aviation safety is compromised Degradation to Level 2 Overconfidence for test-takers Raters Training as a Safety Concern

7. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

8. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications Raters Training as a Safety Concern

9. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Annex 1; Doc 9835; ICAO Rated Speech Samples CD; ICAO Circ 318 Raters Training as a Safety Concern

10. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Operational and/or language specialist expertise Level 5 or Level 6 proficiency Familiarity with Aviation English RATERS CODE OF ETHICS Raters Training as a Safety Concern

11. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

12. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Theory of language testing • ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835,Circ 318 • The ICAO Proficiency Rating Scale and holistic descriptors • Correlating speech samples with the ICAO LPR Scale Raters Training as a Safety Concern

13. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale Raters Training as a Safety Concern

14. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

15. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Certification by national Civil Aviation Authority or by international aviation organization is mandatory Raters Training as a Safety Concern

16. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Is geographical principle necessary? Raters Training as a Safety Concern

17. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Are quotas necessary? Raters Training as a Safety Concern

18. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

19. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different raters but same results Raters Training as a Safety Concern

20. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

21. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different times but same results Raters Training as a Safety Concern

22. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

23. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Raters Training as a Safety Concern

24. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Raters Training as a Safety Concern

25. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Administrative measures to marginal raters Raters Training as a Safety Concern

26. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

27. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters Raters Training as a Safety Concern

28. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Raters Training as a Safety Concern

29. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Practical training Raters Training as a Safety Concern

30. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

31. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN Raters Training as a Safety Concern

32. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Thank you Raters Training as a Safety Concern