slide1
Download
Skip this Video
Download Presentation
Statistics of EBO 2009 Examination Symposium EBO “Training for the Trainers” 116 ème Congrès de la Société Française d’Ophtalmologie (SFO)

Loading in 2 Seconds...

play fullscreen
1 / 31

statistics of ebo 2009 examination symposium ebo training for the trainers 116 me congr s de la soci t fran aise d op - PowerPoint PPT Presentation


  • 332 Views
  • Uploaded on

Statistics of EBO 2009 Examination Symposium EBO “Training for the Trainers” 116 ème Congrès de la Société Française d’Ophtalmologie (SFO). Danny G.P. Mathysen MSc. Biomedical Sciences EBOD Assessment and Executive Officer

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'statistics of ebo 2009 examination symposium ebo training for the trainers 116 me congr s de la soci t fran aise d op' - omer


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Statistics of EBO 2009 ExaminationSymposium EBO “Training for the Trainers”116ème Congrès de la Société Française d’Ophtalmologie (SFO)

Danny G.P. MathysenMSc. Biomedical Sciences

EBOD Assessment and Executive Officer

Antwerp University Hospital, Department of OphthalmologyWilrijkstraat 10, B-2650 Edegem, BelgiumE-mail: [email protected]

slide2

Processing examination results …

Yearly increase of candidates

  • SpeedWell
    • SpeedWell is specialised inorganising medical examinations
    • Optical reader system
      • continuous and yearly increase of applications / interest in EBOD
    • Provided software tools
      • Design of the MCQ answer sheet
      • Design of the Viva Voce mark sheets
      • Statistical analysis output (MultiQuest®) verification of examination results on-site
statistical approaches for ebod
Statistical approaches for EBOD
  • Part I. Written examination (MCQ paper)
    • representing 40 percent of the total candidate score
    • 52 questions, each with 5 true-false items
    • 10 pre-defined topics
    • Available in English (master), French and German (translations)
  • Part II. Oral examination (Viva Voce)
    • representing 60 percent of the total candidate score
    • 4 topics
    • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate
slide5

Descriptive Statistics for EBOD 2009

  • MCQ total scores
    • Range of total scores: 154 – 230
    • Mean ± SD total score: 204.11 ± 13.04

No significant difference!

slide6

Descriptive Statistics for EBOD 2009

No significant differences!

Residents tend to have higher total MCQ scores with lower standard deviations when compared to specialists.

In general there are no statistically significant differences between countries.

descriptive statistics for ebod 2009
Descriptive Statistics for EBOD 2009
  • Careful selection (and modification) of master MCQs
  • Translation of master MCQs (English) to German/French by native-speaking experts in Ophthalmology
  • Verification of correctness of translations by independent EBO Examination Committee members
descriptive statistics for ebod 20098
Descriptive Statistics for EBOD 2009
  • Pre-selecting and controlling of the MCQ paper
    • guarantee that EBOD remains a test in ophthalmology and not a test in language

EBOD is not a language test!

statistical approaches for ebod9
Statistical approaches for EBOD
  • Part I. Written examination (MCQ paper)
    • representing 40 percent of the total candidate score
    • 52 questions, each with 5 true-false items
    • 10 pre-defined topics
    • Available in English (master), French and German (translations)
  • Part II. Oral examination (Viva Voce)
    • representing 60 percent of the total candidate score
    • 4 topics
    • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate
slide10

Statistical analysis of MCQ paper

internal consistencyof EBOD MCQ-test is good

  • Cronbach’s coefficient alpha (r) = 0.78
    • Estimator of the lower bound of the internal consistency (degree to which all MCQs leaves are measuring the same, i.e. knowledge of candidates) of EBOD 2009 (95% CI: 0.75 – 0.81)
slide11

Statistical analysis of MCQ paper

correlation betweenitem and total MCQ score

  • Point biserial correlation coefficient (Rit) = 0.14
    • Estimator of the correlation between the individual item scores Xi (either 0 or 1) and the total MCQ scores Yi (ranging from 154 to 230) of the candidates

-1 0 +1

slide12

Statistical analysis of MCQ paper

Answeredby knowledge

Answeredby guessing

  • Assessment of the degree of difficulty
    • Average P-value ≈ 0.79
      • Indication of items answered incorrectly by guessing ≈ 0.21
      • Estimation of items answered correctly by guessing ≈ 0.21
      • Estimation of percentage of candidates guessing ≈ 0.42OR Estimation of percentage of candidates knowing ≈ 0.58

0 58 79 100

statistical analysis of mcq paper
Statistical analysis of MCQ paper
  • Classical Analysis Methods
    • Cronbach Alpha (internal consistency)
    • Point Biserial Correlation
    • Degree of Difficulty
    • Comparison of item test scores
  • Item-Response analysis
    • Rasch analysis (1-parameter analysis)  items differ only in difficulty
    • 3-Parameter analysis  items differ in difficulty, discriminative power and guess factor
slide14

Statistical analysis of MCQ paper

  • Advantages for EBO candidates of T/F items
    • Reliable in case of translation (English, French, German) choice of language will not result in being (dis)advantaged
    • Accessibility (e.g. dyslexia) not too complicated for candidates
    • Duration of the examination stress level of candidates can be kept to a minimum
    • Relatively easy to process results can be presented on-site
  • Disadvantage for EBO candidates of T/F items
    • Probability of guessing right = 50 % level of weakest candidates is overestimated ( oral examination)
slide15

Statistical analysis of MCQ paper

NEGATIVE MARKINGAT EBOD 20101

Spread of total test scoreswith negative marking

Spread of total test scoreswithout negative marking

  • How to overcome the disadvantages of T/F items?
    • Introduction of negative marking
      • Increase of discriminative power of examination
      • Reduction of guess factor
        • wild guesses will be punished (weakest candidates)
        • guesses by reasoning (partial knowledge) will be rewarded

-130 0 260

slide16

Statistical analysis of MCQ paper

NEGATIVE MARKINGDOES NOT INFLUENCEPASS RATE!

  • Does negative marking influences the pass rate?
    • Score of +1 in case (only) the correct answer is indicated
    • Score of -0.5 in case the incorrect answer or nothing is indicated
    • Score of 0 in case the “D”-option (don’t know) is indicated
    • Score conversion (pass mark = 6) (formula above)
    • Other marks are derived
definition of pass mark
Definition of pass mark

Synonyms: standard, cutpoint

A pass mark is a special score that serves as boundary between those who perform well enough and those who do not

0 100

How to set pass marks? Reaching a consensus rather than obtaining a scientifically correct solution

importance of pass marks
Importance of pass marks

The purpose of an examination is to select the group of candidates that perform well enough (pass) and to eliminate the group of candidates that do not perform well enough (fail)

In order to achieve this goal, a (limited) number of questions are presented to the candidates

The discriminative power of the examination will depend on the validity of the questions used

validity of questions
Validity of questions

Degree of difficulty of questions

Can be assessed by calculating the P-value (i.e. percentage of candidates answering correctly)

Degree of discriminative power of questions

Objective measurement of the degree to which the question is able to discriminate strong from weak candidates

Can be assessed by calculating the Rit/Rir value (correlation of question score to total examination score)

Thumb rule:

Avoid questions with P-value above 0.90 or below 0.10

Thumb rule:

Avoid questions with Rit-value below 0.20

types of pass marks
Types of pass marks

“absolute” pass mark (criterion-reference)

expressed as a number (e.g. 70 correct responses) of test questions

expressed as a percentage (e.g. 70 % correct responses) of test questions

“relative” pass mark (norm-reference)

expressed as a number (e.g. 50 best performers) of examinees

expressed as a percentage (e.g. top 20 % performers) of examinees

  • how to determine reasonable criteria for candidates?
  • flexibility in case you are not familiar with the technique

Validation of questions

  • number of candidates ≥ 40
  • candidates have to take the test on an individual basis
slide21

Statistical analysis of MCQ paper

NEGATIVE MARKINGDOES NOT INFLUENCETHE PASS RATE !!!

  • Does negative marking influence the pass rate?
    • Score of +1 in case (only) the correct answer is indicated
    • Score of -0.5 in case the incorrect answer or nothing is indicated
    • Score of 0 in case the “D”-option (don’t know) is indicated
    • Score conversion (pass mark = 6) (formula above)
    • Other marks are derived
statistical approaches for ebod22
Statistical approaches for EBOD
  • Part I. Written examination (MCQ paper)
    • representing 40 percent of the total candidate score
    • 52 questions, each with 5 true-false items
    • 10 pre-defined topics
    • Available in English (master), French and German (translations)
  • Part II. Oral examination (Viva Voce)
    • representing 60 percent of the total candidate score
    • 4 topics
    • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate
statistical approaches for ebod23
Statistical approaches for EBOD
  • Examiners
    • Careful pre-selection
    • Clear instructions before examination
    • Different questions
    • Different languages
  • Nevertheless…
    • MCQ and Viva Voce scores are well correlated
    • No (dis)advantage for candidates to be assigned to any specific jury
slide24

Statistical approaches for EBOD

EBOD scores are high!

statistical approaches for ebod25
Statistical approaches for EBOD
  • Part I. Written examination (MCQ paper)
    • representing 40 percent of the total candidate score
    • 52 questions, each with 5 true-false items
    • 10 pre-defined topics
    • Available in English (master), French and German (translations)
  • Part II. Oral examination (Viva Voce)
    • representing 60 percent of the total candidate score
    • 4 topics
    • Available in English, French, German (basic languages) and(whenever possible) in native language of the candidate
slide26

Statistical approaches for EBOD

EBOD scores arecomparable for MCQand Viva Voce!

Residents have higher MCQ and Viva Voce scores with lower standard deviations when compared to specialists.

slide27

Statistical approaches for EBOD

Success rate of EBOD is much higher ascompared to other medical specialties (60-70 %)

EBOD success rate is quite stable over the years and quite high as the level of candidates usually tends to be good.

18 Residents (out of 220: 8.2%)and 14 specialists (out of 88: 15.9 %)failed at EBOD 2009. As there were 308 candidates the general failure rate was 10.4 %.

slide28

General Conclusions

Careful monitoring of EBOD

Careful validation of EBOD

Reliable results of EBOD

Stable results of EBOD over the years

Result of careful pre- and post-assessment of EBOD

slide29

General Conclusions

Publication on EBO website

slide30

General Conclusions

  • Presentation at scientific meetings
    • Société française d’Ophtalmologie (SFO 2009-2010)
    • European Society of Ophthalmology (SOE 2009)
    • Council for European Specialist Medical Examinations(CESMA 2009-2010)
    • International Association for Medical Education (AMEE 2010)
  • Publication in peer-reviewed journal
    • Manuscripts in preparation
ad