1 / 22

The Role of Formal Assessment in Medical Regulation IAMRA 2010

The Role of Formal Assessment in Medical Regulation IAMRA 2010. Stephen I. Schabel MD , FACR Professor of Radiology Medical University of South Carolina Formerly of. SC SBME. USMLEComposite Step3 Step1. FSMB-BOD.

katoka
Download Presentation

The Role of Formal Assessment in Medical Regulation IAMRA 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Role of Formal Assessment in Medical Regulation IAMRA 2010 Stephen I. Schabel MD , FACR Professor of Radiology Medical University of South Carolina Formerly of SC SBME USMLEComposite Step3 Step1 FSMB-BOD ECFMG NBME

  2. Medical Regulators • USA = States – Medical licensure a state function • “State funded” – by licensure fees – usually most taken by state government • Charge – PROTECT the PUBLIC – from substandard physicians • NO resources to produce independent knowledge examinations • Therefore have to trust SOMEONE

  3. Initial Medical Licensure in US • Education - M.D. , D.O. ,IMG • Examination of Knowledge – USMLE • Post Graduate Training – ACGME ( 2-3 Years) • External Check – FSMB / State

  4. United States Medical LicensureExaminations - History/Evolution • Independent State Licenses – circa 1900 • Independent State Examinations – FSMB 1912 -reciprocity • Medical Education Quality – marked variation – Flexner 1910 • Post Graduate Training – contact driven – great variation pre ABMS 1933 • NBME – 1915 – badge of Honor/ excellence (LCME only) • ECFMG 1957 – Federal Charter VQE – NBME prepared • FLEX – 1960’s – ( NBME prepared ) states used (IMG /US) • USMLE – 1994 – single pathway to licensure

  5. Why? Equity of Common Standards 1964 Assessed: 20 state medical licensing exams Findings: Concerns for validity issues and content quality George Miller, MD Office of Research in Medical Education University of Illinois College of Medicine 1965 Reviewed fail rates for state board examinations, 1954-1964 • 3 states failed no one all 10 years • 2 states failed no one 9 out of 10 years • 4 states failed no one 8 out of 10 years Robert Derbyshire, MD Federation Bulletin, April 1965

  6. United States – An Egalitarian Country • As Americans we want to believe that an individuals effort and ability are the only limitations to eventual success • At least by providing a pathway for success by ability we are closer to our ideal • In Medicine Individual Ability and Commitment is the only limitation to success

  7. US a Country of Volunteers • Self regulation an important principal • Professionals historically willing to participate • Medical Board Members , FSMB, ECFMG, and USMLE/NBME – “The National Faculty of Medicine”

  8. Why MCQ’s • Psychometrically most valid and cost effective ( still expensive) – A Type • We are a country of MCQ test takers • Other formats are being explored

  9. Formal Assessments in Medicine in United States – A National Standard • You may think Harvard is better than University of South Carolina – but if a SC student get a +3 SD score – you know they are smart • MCAT – Medical College Assessment Test (SAT) • USMLE – Unites States Medical Licensure Examination • ABMS – American Board of Medical Specialties – certification and MOC • SPEX – Special Purpose Examination • We have learned to LOVE standardized Tests

  10. ABMS Certification • No state requires ABMS certification for licensure – all states accept it as a mark of competence • Many Hospitals DO require ABMS certification for staff privileges • More and More payers require ABMS certification for reimbursement

  11. FLEX NBME-USMLE – Why a Single Exam We Don’t Want To Be Sued • Prior to 1994 IMGs- had to take FLEX - 60% passed • Most US LCME students took NBME -99% passed but could take FLEX • FSMB was sued – unfair – FLEX must be a harder • USMLE - % unchanged

  12. As Licensure Board/Residency Director/PatientWhat Can a Regulator Believe? • What the doctor says or writes? • Patient testimonials? • The letters or recommendation / Dean’s letter / Program Directors Letter • Medical School Transcript • License in Another State or Country • USMLE – ABMS …..

  13. Still Needed? • Who can you Trust? • Every student in an LCME accredited medical school who takes USMLE Step 1 and 2 – has been certified as competent by their medical school’s Dean of Students and faculty as competent – 5% + are judged substandard • 100% of candidates for Step 3 of USMLE have an MD degree and 3% + are found inadequate • Not the Deans of Students of Medical Schools! • Every candidate for ABMS Board Certifying Examinations has been recommended by their program director as competent and many (5-30%) are found inadequate. • Not Program Directors!

  14. Lake Wobegon Effect • 68% of U. Nebraska Faculty rate themselves in top 25% • 87% of Stanford MBA students rate themselves above mean • 70% of US college bound students rate themselves above average – 25% in top 1 % • 93% of US drivers say their ability in top 50%

  15. Medical Licensure ExaminationsWhat Would You Really Like • Accurate Measure of Knowledge needed to practice ( as predictor of competence) • Quality – secure ,current and correct content , high fidelity • Complete but not Excessive – 39 Hours total • Inexpensive (US $2700)/ Convenient (CBT- 100’s centers/CK 5 centers) • Fair – free of bias toward women, minorities, regions , medical schools , IMGs ( English) • Portability/ Universal Acceptance

  16. Why? Provides data for evidence-based education/regulation Tamblyn, et al. JAMA, December 2002

  17. USMLE – A Model of Medical Evaluation • Single Examination for all M.D.s – open to all Medical Graduates (M.D. and D.O.) – 140,000/year • Single Standard - everyone held to the same one • Fixed Standard – all could pass or fail • Three Steps

  18. USMLE 3 Steps • Step 1 – after basic sciences – fundamental scientific knowledge necessary to beginning clinical education – 1 day MCQ CBT • Step 2 CK – after clinical rotations Medical school – clinical knowledge necessary to begin supervised Post Graduate training – 1 day MCQ CBT • Step 2 CS – clinical skills , communication necessary to begin supervised practice – 1 day multi station OSKE • Step 3 – knowledge necessary to begin unsupervised practice ,management – 2 days CBT MCQ

  19. Examinations of the 6 Competencies • ACGME-ABMS- Medical school education – accepted 6 competencies of Physicians • Patient Care • Medical Knowledge • Practice based Learning • Interpersonal and Communication Skills • Professionalism • Systems based Practice • USMLE MCQs- first 2 – Step 2 CS –communication • Others very difficult – evaluation methodologies either unavailable or lesser psychometrically weaker – longer / more expensive • Research ongoing

  20. Why USMLE Makes Sense For US • Economies of scale – states and even regions lack adequate recourses • National Exam -attracts National Faculty of Medicine - Schools can examine curriculum ,teaching and student performance and testing strategies • Mobile physician and patient populations

  21. Why Not Accept International Certifications • Little First Hand Knowledge – Unknown Quality • Accept ONE have to accept ALL • Medicine Elsewhere Different than US – USMLE in France – 10% of questions answered differently –PPD reactivity and BCG

More Related