1 / 17

Standard Setting for a Performance-Based Examination for Medical Licensure

Standard Setting for a Performance-Based Examination for Medical Licensure. Sydney M. Smee Medical Council of Canada. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona. MCC Qualifying Examination Part II. OSCE format - 12 short stations

Download Presentation

Standard Setting for a Performance-Based Examination for Medical Licensure

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Standard Setting for a Performance-Based Examination for Medical Licensure Sydney M. Smee Medical Council of Canada Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  2. MCC Qualifying Examination Part II • OSCE format - 12 short stations • 5 or 10 minutes per patient encounter • Physicians observe and score performance • Required for medical licensure in Canada • Prerequisites • Passed MCCQE Part I (Knowledge & Clinical reasoning) • Completed 12 months of post-graduate clinical training • Pass/Fail criterion-referenced examination • Multi-site administration - twice per year • Overall fail rate 10%-30% • Implemented 1992 Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  3. Why do it? • Requested by licensing authorities, largely in response to two issues: • Increase in complaints, many centered around communication skills. • Public accountability - OSCE to serve as an “audit” of training of all candidates seeking licensure in Canada. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  4. Blueprint Considerations • Four domains *History-taking *Patient Interaction *Physical Examination *Management • Multi-disciplinary / multi-system content • Patient demographics • Two formats *5+5 couplets & 10 minute • Each case based on an MCC Objective Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  5. Standard for MCCQE Part II • Acceptably competent for entry to independent practice • Conjunctive standard • Pass by total score AND • Pass by minimum number of stations • High performance in a few stations does not compensate for overall poor performance • Just passing enough stations does not compensate for overall poor performance Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  6. Translating a Standard to a Pass Mark • Pilot exam: Ebel method • Items rated for relevance and importance • Pass based on most relevant and important items • Failed 40% • First two administrations: Angoff method • Estimated score for the minimally competent candidate • Pass based on average of estimates per instrument • Pass marks varied more than the test committee liked • Test committee did not like the task • 1994: Adopted borderline group method Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  7. Physicians as Scorers Three Assumptions: • Clinicians do not require training to judge candidate behaviour according to checklists for basic clinical skills • Most clinicians can make expert judgments about candidate performance • Being judged by clinicians is vital for a high-stakes examination Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  8. Physicians as Standard Setters Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  9. Global Rating Question • Did the candidate respond satisfactorily to the needs/problems presented by this patient? • Borderline Unsatisfactory • Unsatisfactory • Inferior • Borderline Satisfactory • Good • Excellent Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  10. Numbers.... • 1,000-2,200 candidates per administration • Examiners each observe 16-32 candidates • 20-60 examiners per case • Number of candidates identified as borderline per case ranges from 150-500 • Collect >99% of data for global rating item Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  11. Modified Borderline Group Method • Examiners (content experts) identify borderline candidates based on the 6-point scale • Scores of borderline candidates define performance that “describes” the pass standard • Examiner judgments are translated into a pass mark by taking the mean score for the borderline candidates for each case Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  12. Pass Marks by Case Across Exams • Challenge to assess pass marks over multiple administrations • Scoring instruments are revised post-exam • Rating scale items have been revised • Rating scale items have been added to cases • As competency and difficulty of cases changes, so do cut scores Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  13. Setting Total Exam Pass Mark • Pass marks for cases are summed • Add one standard error of measure (3.2% ) • Pass mark falls between 1 to 1.5 SD below mean score • Station performance is reviewed by Central Examination Committee • Then the standard for the number of stations passed is set • Standard has been 8/12 since 2000 Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  14. Outcomes • 15,331 candidates became eligible in 2000 – 2005 • 6,099 have yet to attempt MCCQE Part II • 8,514 have passed • 718 or 7.7% failed • 2,243 candidates were eligible prior to 2000 and also took MCCQE Part II in 2000 – 2005 • 2,166 have passed • 77 or 3.4% failed and are likely out of the system • Fail rates do not reflect impact on repeat takers • Focused hundreds of candidates on remediation Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  15. Limitation • Current approach is easy to implement but it relies upon • Large number of standard setters per case • Large number of test takers in borderline group • Smaller numbers would lead to more effort • Increase training of examiners • Impose stricter selection criteria on standard setters Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  16. What’s ahead? • Increasing number of candidates to be assessed each year • Modifications to the administration are needed • Predictive validity study currently in progress • Use non-physician examiners? • Which type of cases, who sets standard? • Add more administrations? • Case development / challenge of piloting content Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  17. Medical Council of Canada Ottawa Sydney M. Smee, M.Ed. Manager, MCCQE Part II www.mcc.ca Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

More Related