What is EBMA?

1. Preliminary psychometric properties of a novel test designed to assess application of medical knowledge in European countries Carlos Fernando Collares, Maastricht University Adrian Freeman, University of Exeter Lesley Southgate, St George’s Hospital Medical School René Tio, University of Groningen Annemarie Camp, Maastricht University Cees P. M. van der Vleuten, Maastricht University www.ebma.eu.com

2. What is EBMA? • Non-profit organization formed by a group of European professionals who have expertise in assessment and/or have leadership roles in universities, or other bodies concerned with medical education and training.  • Not a licensing body • Focus on Assessment FORlearning

3. Why EBMA? • Promotion of best assessment practices • Optimal feedback about learning needs • Support individuals and institutions • European standards for assessment programmes • European framework for quality assurance in healthcare • Patient safety • Physician mobility • Flexibility for local, national interests

4. EBMA members • Belgium: Ghent University • Denmark: University of Copenhagen • Finland: University of Helsinki • Germany: University of Heidelberg • Italy: Catholic University of Rome • Netherlands: University Medical Center Groningen • Netherlands: Maastricht University • Poland: Jagiellonian University • Portugal: University of Minho • UK: Keele University • UK: University of Exeter • UK: Plymouth University

5. European Knowledge Test (EKT) • Voluntary written exam • 200 multiple-choice questions, all scenario-based • Focus on application of medical knowledge at the level of a recently graduated physician • Carefully written and reviewed by EBMA founding members • Expert-reviewed blueprint, compatible with EBMA International Progress Test (IPT) • Angoff-based cutscore (ICC = 0,952) • Not a licensing exam • Support for medical students and graduates in Europe • Enables insight into learning needs • Interactive online feedback tool

6. Interactive online feedback

7. International assessment initiatives are increasingly common. Interpreting results might be a very challenging endeavour.

8. Measurementinvariance • A necessaryconditionformeaningful, validcomparisons. • Same construct, similarlymeasured, across different groups. • Differential item functioning(DIF) is a methodtostudymeasurementinvariancebetweentwogroups. • Measurement alignment: method to study measurement invariance between several groups; • Whenmeasurement non-invariance is detected in an item, it is a strong suggestionthatsomethingelsebesidestheintended construct (e.g. application of medicalknowledge) is beingmeasured.

9. Objectives • This study aims to describe the preliminary psychometric findings of the pilot administrations of the European Knowledge Test (EKT) in terms of: • Reliability: Cronbach’s alpha, split-half correlations and IRT based conditional reliability • Validity: item fit statistics and measurement invariance (DIF and measurement alignment).  • Item discrimination and distractor analysis

10. Participants • Ghent University: 100 students (18 in the 1st pilot; 82 in the second pilot) • University of Copenhagen: 16 students • University of Heidelberg: 3 students • University Medical Center Groningen: 16 students • Maastricht University: 19 students • University of Łódź: 19 students • University of Minho: 42 students • TOTAL: 215 participating students

11. Pilots • 1stpilot phase (February – March 2014) in 4 institutions. • 2ndpilot phase (October 2014 – February 2015) in 3 institutions and an additional pilot in one institution that had already participated in the 1st pilot phase.

12. Data analysis • Classical test theory and Item Response Theory (Rasch model) • Compliance with unidimensionality and local independence • Measurement alignment • Software: WINSTEPS, XCALIBRE, ITEMAN and Mplus.

13. RESULTS

14. Reliability • Mean individual precision estimates (Rasch model) = 0,89 (SD = 0,01) • Cronbach’s alpha = 0,90 • Split-half Pearson correlations: 0,80 (random); 0,67 (first-last); 0,84 (odd-even) • Spearman-Brown correlations: 0,89 (random); 0,80 (first-last); 0,91 (odd-even)

15. Item fit • Infit and outfit mean-squares calculations revealed “overfit”, suggesting “redundant” items: • No items with infit or outfit above 1,3. • Both infit and outfit statistics below 0,7: 23 items (11,5%) • Only infit below 0,7: 1 item (0,5%) • Only outfit below 0,7: 2 items (1,0%)

16. Measurementinvariance • Analysis between the two pilot phasesfailedto show any items withdifferential item functioning(DIF) basedeither on the Rasch model or classical test theory. • MeasurementalignmentusingBayesianestimationresulted in 49 non-invariant items (24,5%): • In one group: 42 (21,0%); • In two groups: 3 (1,5%) • In three groups: 3 (1,5%) • In four groups: 0 (0,0%) • In five groups: 1 (0,5%) • Most non-invariant items were found in the same group (N = 25; 12,5%).

17. Item discrimination • Item-totalbiserialcorrelationswerenegative in 11 items (5,5%). • In 33 items (16,5%), at leastone of the distractors had an item-totalcorrelationhigherthan the correct answer. • Sample sizeincrease has consistentlyreduced the number of items withdysfunctionaldiscrimination.

18. Post-test review • Items flagged for measurement non-invariance or dysfunctional discrimination were carefully reviewed in an attempt to identify the underlying motives. • Differences in definitions of concepts. • Different national guidelines. • Cross-cultural differences in prescribing behavior.

19. Conclusionsand take-home messages • The EKT pilot presented a good degree of reliability and validity based on internal structure evidences. • Inclusion of students from more schools and countries will enable a more representative reference group. • Further exploration of the psychometric properties and the educational utility of the EKT seems to be worthy. • Psychometric analyses and careful post-test review are a valuable tool to the further enhance the quality of the EKT. • EKT can become a useful assessment tool for European medical students and schools.

20. http://www.ebma.eu.com http://twitter.com/EBMAorg http://www.facebook.com/ebma.eu THANK YOU!