1 / 31

Statistics of EBO 2009 Examination Symposium EBO “Training for the Trainers” 116 ème Congrès de la Société Française d’

Statistics of EBO 2009 Examination Symposium EBO “Training for the Trainers” 116 ème Congrès de la Société Française d’Ophtalmologie (SFO). Danny G.P. Mathysen MSc. Biomedical Sciences EBOD Assessment and Executive Officer

omer
Download Presentation

Statistics of EBO 2009 Examination Symposium EBO “Training for the Trainers” 116 ème Congrès de la Société Française d’

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistics of EBO 2009 ExaminationSymposium EBO “Training for the Trainers”116ème Congrès de la Société Française d’Ophtalmologie (SFO) Danny G.P. MathysenMSc. Biomedical Sciences EBOD Assessment and Executive Officer Antwerp University Hospital, Department of OphthalmologyWilrijkstraat 10, B-2650 Edegem, BelgiumE-mail: danny.mathysen@uza.be

  2. Processing examination results … Yearly increase of candidates • SpeedWell • SpeedWell is specialised inorganising medical examinations • Optical reader system • continuous and yearly increase of applications / interest in EBOD • Provided software tools • Design of the MCQ answer sheet • Design of the Viva Voce mark sheets • Statistical analysis output (MultiQuest®) verification of examination results on-site

  3. Statistical approaches for EBOD • Part I. Written examination (MCQ paper) • representing 40 percent of the total candidate score • 52 questions, each with 5 true-false items • 10 pre-defined topics • Available in English (master), French and German (translations) • Part II. Oral examination (Viva Voce) • representing 60 percent of the total candidate score • 4 topics • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate

  4. Descriptive Statistics for EBOD 2009 Many EU countries apply

  5. Descriptive Statistics for EBOD 2009 • MCQ total scores • Range of total scores: 154 – 230 • Mean ± SD total score: 204.11 ± 13.04 No significant difference!

  6. Descriptive Statistics for EBOD 2009 No significant differences! Residents tend to have higher total MCQ scores with lower standard deviations when compared to specialists. In general there are no statistically significant differences between countries.

  7. Descriptive Statistics for EBOD 2009 • Careful selection (and modification) of master MCQs • Translation of master MCQs (English) to German/French by native-speaking experts in Ophthalmology • Verification of correctness of translations by independent EBO Examination Committee members

  8. Descriptive Statistics for EBOD 2009 • Pre-selecting and controlling of the MCQ paper • guarantee that EBOD remains a test in ophthalmology and not a test in language EBOD is not a language test!

  9. Statistical approaches for EBOD • Part I. Written examination (MCQ paper) • representing 40 percent of the total candidate score • 52 questions, each with 5 true-false items • 10 pre-defined topics • Available in English (master), French and German (translations) • Part II. Oral examination (Viva Voce) • representing 60 percent of the total candidate score • 4 topics • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate

  10. Statistical analysis of MCQ paper internal consistencyof EBOD MCQ-test is good • Cronbach’s coefficient alpha (r) = 0.78 • Estimator of the lower bound of the internal consistency (degree to which all MCQs leaves are measuring the same, i.e. knowledge of candidates) of EBOD 2009 (95% CI: 0.75 – 0.81)

  11. Statistical analysis of MCQ paper correlation betweenitem and total MCQ score • Point biserial correlation coefficient (Rit) = 0.14 • Estimator of the correlation between the individual item scores Xi (either 0 or 1) and the total MCQ scores Yi (ranging from 154 to 230) of the candidates -1 0 +1

  12. Statistical analysis of MCQ paper Answeredby knowledge Answeredby guessing • Assessment of the degree of difficulty • Average P-value ≈ 0.79 • Indication of items answered incorrectly by guessing ≈ 0.21 • Estimation of items answered correctly by guessing ≈ 0.21 • Estimation of percentage of candidates guessing ≈ 0.42OR Estimation of percentage of candidates knowing ≈ 0.58 0 58 79 100

  13. Statistical analysis of MCQ paper • Classical Analysis Methods • Cronbach Alpha (internal consistency) • Point Biserial Correlation • Degree of Difficulty • Comparison of item test scores • Item-Response analysis • Rasch analysis (1-parameter analysis)  items differ only in difficulty • 3-Parameter analysis  items differ in difficulty, discriminative power and guess factor

  14. Statistical analysis of MCQ paper • Advantages for EBO candidates of T/F items • Reliable in case of translation (English, French, German) choice of language will not result in being (dis)advantaged • Accessibility (e.g. dyslexia) not too complicated for candidates • Duration of the examination stress level of candidates can be kept to a minimum • Relatively easy to process results can be presented on-site • Disadvantage for EBO candidates of T/F items • Probability of guessing right = 50 % level of weakest candidates is overestimated ( oral examination)

  15. Statistical analysis of MCQ paper NEGATIVE MARKINGAT EBOD 20101 Spread of total test scoreswith negative marking Spread of total test scoreswithout negative marking • How to overcome the disadvantages of T/F items? • Introduction of negative marking • Increase of discriminative power of examination • Reduction of guess factor • wild guesses will be punished (weakest candidates) • guesses by reasoning (partial knowledge) will be rewarded -130 0 260

  16. Statistical analysis of MCQ paper NEGATIVE MARKINGDOES NOT INFLUENCEPASS RATE! • Does negative marking influences the pass rate? • Score of +1 in case (only) the correct answer is indicated • Score of -0.5 in case the incorrect answer or nothing is indicated • Score of 0 in case the “D”-option (don’t know) is indicated • Score conversion (pass mark = 6) (formula above) • Other marks are derived

  17. Definition of pass mark Synonyms: standard, cutpoint A pass mark is a special score that serves as boundary between those who perform well enough and those who do not 0 100 How to set pass marks? Reaching a consensus rather than obtaining a scientifically correct solution

  18. Importance of pass marks The purpose of an examination is to select the group of candidates that perform well enough (pass) and to eliminate the group of candidates that do not perform well enough (fail) In order to achieve this goal, a (limited) number of questions are presented to the candidates The discriminative power of the examination will depend on the validity of the questions used

  19. Validity of questions Degree of difficulty of questions Can be assessed by calculating the P-value (i.e. percentage of candidates answering correctly) Degree of discriminative power of questions Objective measurement of the degree to which the question is able to discriminate strong from weak candidates Can be assessed by calculating the Rit/Rir value (correlation of question score to total examination score) Thumb rule: Avoid questions with P-value above 0.90 or below 0.10 Thumb rule: Avoid questions with Rit-value below 0.20

  20. Types of pass marks “absolute” pass mark (criterion-reference) expressed as a number (e.g. 70 correct responses) of test questions expressed as a percentage (e.g. 70 % correct responses) of test questions “relative” pass mark (norm-reference) expressed as a number (e.g. 50 best performers) of examinees expressed as a percentage (e.g. top 20 % performers) of examinees • how to determine reasonable criteria for candidates? • flexibility in case you are not familiar with the technique Validation of questions • number of candidates ≥ 40 • candidates have to take the test on an individual basis

  21. Statistical analysis of MCQ paper NEGATIVE MARKINGDOES NOT INFLUENCETHE PASS RATE !!! • Does negative marking influence the pass rate? • Score of +1 in case (only) the correct answer is indicated • Score of -0.5 in case the incorrect answer or nothing is indicated • Score of 0 in case the “D”-option (don’t know) is indicated • Score conversion (pass mark = 6) (formula above) • Other marks are derived

  22. Statistical approaches for EBOD • Part I. Written examination (MCQ paper) • representing 40 percent of the total candidate score • 52 questions, each with 5 true-false items • 10 pre-defined topics • Available in English (master), French and German (translations) • Part II. Oral examination (Viva Voce) • representing 60 percent of the total candidate score • 4 topics • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate

  23. Statistical approaches for EBOD • Examiners • Careful pre-selection • Clear instructions before examination • Different questions • Different languages • Nevertheless… • MCQ and Viva Voce scores are well correlated • No (dis)advantage for candidates to be assigned to any specific jury

  24. Statistical approaches for EBOD EBOD scores are high!

  25. Statistical approaches for EBOD • Part I. Written examination (MCQ paper) • representing 40 percent of the total candidate score • 52 questions, each with 5 true-false items • 10 pre-defined topics • Available in English (master), French and German (translations) • Part II. Oral examination (Viva Voce) • representing 60 percent of the total candidate score • 4 topics • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate

  26. Statistical approaches for EBOD EBOD scores arecomparable for MCQand Viva Voce! Residents have higher MCQ and Viva Voce scores with lower standard deviations when compared to specialists.

  27. Statistical approaches for EBOD Success rate of EBOD is much higher ascompared to other medical specialties (60-70 %) EBOD success rate is quite stable over the years and quite high as the level of candidates usually tends to be good. 18 Residents (out of 220: 8.2%)and 14 specialists (out of 88: 15.9 %)failed at EBOD 2009. As there were 308 candidates the general failure rate was 10.4 %.

  28. General Conclusions Careful monitoring of EBOD Careful validation of EBOD Reliable results of EBOD Stable results of EBOD over the years Result of careful pre- and post-assessment of EBOD

  29. General Conclusions Publication on EBO website

  30. General Conclusions • Presentation at scientific meetings • Société française d’Ophtalmologie (SFO 2009-2010) • European Society of Ophthalmology (SOE 2009) • Council for European Specialist Medical Examinations(CESMA 2009-2010) • International Association for Medical Education (AMEE 2010) • Publication in peer-reviewed journal • Manuscripts in preparation

  31. General Conclusions

More Related