Usmle step 2 clinical skills l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 32

“USMLE Step 2 Clinical Skills” PowerPoint PPT Presentation


  • 228 Views
  • Uploaded on
  • Presentation posted in: General

“USMLE Step 2 Clinical Skills”. Richard E. Hawkins, MD National Board of Medical Examiners. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona. Presentation Structure. Overview of USMLE Step 2 CS Measures to ensure fairness and consistency.

Download Presentation

“USMLE Step 2 Clinical Skills”

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Usmle step 2 clinical skills l.jpg

“USMLE Step 2 Clinical Skills”

Richard E. Hawkins, MD

National Board of Medical Examiners

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Presentation structure l.jpg

Presentation Structure

  • Overview of USMLE Step 2 CS

  • Measures to ensure fairness and consistency

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Overview of usmle step 2 cs l.jpg

Overview of USMLE Step 2 CS

Purpose and Delivery Model

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle purpose l.jpg

USMLE: Purpose

  • Three step examination system primarily designed to support the medical licensing process

  • Jointly sponsored by

    • Federation of State Medical Boards (FSMB)

    • National Board of Medical Examiners (NBME)

  • Step 2 CS: collaboration with ECFMG

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle purpose5 l.jpg

USMLE: Purpose

  • Overall purpose

    • Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that constitute the basis of safe and effective patient care

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle purpose6 l.jpg

USMLE: Purpose

  • Overall purpose

    • Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that constitute the basis of safe and effective patient care

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle purpose7 l.jpg

USMLE: Purpose

  • Patient-centered Skills

    • History taking / Physical examination

    • Communication and Interpersonal Skills

    • Medical Record Documentation

  • Important for safe and effective patient care

    • History/PE  Diagnosis and Management

    • Communication  Patient Health Outcomes

    • Medical Record  Errors and Patient Safety

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle step 2 cs l.jpg

USMLE Step 2 CS

  • Enhancement to USMLE Step 2

  • USMLE Step 2 – components:

    • Clinical knowledge (CK)

    • Clinical skills (CS)

  • Standard for Step 2 – appropriate for entry into postgraduate training

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle step 2 cs logistics l.jpg

USMLE Step 2 CS: Logistics

  • Regional delivery model

    • Optimal combination of convenience, cost-efficiency and standardization

  • Five regional test centers across US

  • Projected examine volume – 30,000+ / year

  • Individual center capacity

    • 3 examinations / day (33 examinees); up to 7 days/week

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Requirements of a high stakes performance based examination l.jpg

Requirements of a High StakesPerformance-based Examination

Ensuring Fairness and Consistency

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Threats to validity and reliability l.jpg

Threats to Validity and Reliability

  • Content and/or tasks not relevant or realistic

  • Individual test forms vary in content coverage

  • Scoring methods not appropriate for skills tested

  • Inconsistency in SP portrayal and scoring

    • Between cases, across sites, over time

  • Level of difficulty of cases / exams inconsistent

  • Standard setting approach appropriate

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle step 2 cs content l.jpg

USMLE Step 2 CS: Content

  • Step 2 CS Blueprint

    • Defines content categories

    • In meeting blueprint specifications, each test form provides:

      • Adequate sampling of content domain

      • Comparable content between test forms

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Usmle step 2 cs content13 l.jpg

USMLE Step 2 CS: Content

Blueprint Content Categories:

  • Common and important medical problems / patient presentations

  • Acuity

  • Age

  • Gender

  • Race / ethnicity

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Slide14 l.jpg

Case Content

Cardiovascular

Respiratory Gastrointestinal

Musculoskeletal

Constitutional

Neurological

Psychiatric Genitourinary

Women’s health

Unclassified / multi-system

Case Acuity

Acute

Subacute / Chronic

Test Form

Patient age

Age less than 18

Age 18 – 44

Age 45 – 64

Age 65 +

Patient Gender

Male

Female

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Step 2 cs structure tasks l.jpg

Step 2 CS: Structure & Tasks

  • 12 patient encounters

  • 15 min. for encounter / 10 min. for patient note

  • Each encounter:

    • Elicit pertinent history,

      Perform appropriate physical examination,

      Communicate effectively

    • Document:

      • Findings from the history and physical

      • Diagnostic impression / Further work-up

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Step 2 cs score components l.jpg

Step 2 CS: Score Components

  • Integrated Clinical Encounter (ICE)

    • Data gathering: history and physical exam

    • Patient note

  • Communication / Interpersonal Skills (CIS)

    • Gathering information; sharing information; manner & rapport

  • Spoken English Proficiency (SEP)

    • Listener effort, examinee pronunciation / word choice

Presented at the 2005 CLEAR Annual Conference

September 15-17 Phoenix, Arizona


Scoring approach ice l.jpg

Scoring Approach: ICE

  • Data gathering (Hx / PE)

    • Dichotomous checklists completed by SPs

      • +

  • Patient note

    • Physician raters using holistic methods

      • =

  • Clinical Process + Clinical Outcome Measure

  • Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Case and checklist development l.jpg

    Case and Checklist Development

    • Iterative Process

      • Involves test committees

      • Focus on clinical presentation

      • Checklists “limited”

      • Encounters with SPs

      • Appropriateness of content and difficulty

    • Stepwise progression through pilot and calibration stages

      • Validation / refinement via review of examinee performance data

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Patient note l.jpg

    Patient Note

    • Holistic scoring

    • Rater training

      • General policies

      • Case specific:

        • Consensus development on key features

      • Calibration phase

      • Quality Assurance

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Derivation of communication interpersonal skills cis scale l.jpg

    Derivation of Communication / Interpersonal Skills (CIS) Scale

    • Initial instrument – validated ECFMG scale

    • Reviewed against national consensus recommendations and commonly used scales

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Cis subscales l.jpg

    CIS Subscales

    • Data gathering skills

      • Open-ended questions, transitional statements, not interrupting the patient

    • Information sharing skills

      • Responsiveness to patient questions/concerns, provision of counseling when appropriate, avoidance of jargon

    • Personal manner and rapport

      • Expression of interest in the impact of the illness, concern for patient comfort and modesty

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Step 2 cs scoring l.jpg

    Step 2 CS: Scoring

    • Reporting is Pass or Fail only

    • Examinees must pass all three subcomponents

    • Feedback to examinees

      • Performance report – overall and subcomponent outcomes

      • Failing examinees only – graphical profiles

        • Intended to show relative strengths and weaknesses

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Slide23 l.jpg

    Sample PerformanceReport

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Sample performance profile l.jpg

    Sample Performance Profile

    Lower Performance

    Borderline

    Performance

    Higher Performance

    Integrated Clinical Encounter

    Data Gathering

    Patient Note

    Communication / Interpersonal Skiills

    Spoken English Proficiency

    XXXXXXXXXXXXX

    XXXXXXXXXXXXXXXX

    XXXXXXXXXXXXXXXX

    XXXXXXXXXXXXXX

    XXXXXXXXXXXXX

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Threats to validity and reliability25 l.jpg

    Threats to Validity and Reliability

    • Content and/or tasks not relevant or realistic

    • Individual test forms vary in content coverage

    • Scoring methods not appropriate for skills tested

    • Inconsistency in SP portrayal and scoring

      • Between cases, across sites, over time

    • Level of difficulty of cases / exams inconsistent

    • Standard setting approach appropriate

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Ensuring fairness and consistency l.jpg

    Ensuring Fairness and Consistency

    • Rigorous SP (and SP trainer) training

    • Meticulous attention to quality assurance

    • Application of equating procedures

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Sp and sp trainer training l.jpg

    Standardized Patients

    Rigorous training – generic and case specific

    Electronic delivery (“Ecase”) of case materials

    Sign off process for SPs – Criteria : # portrayals, tests

    SP Trainers

    Adherence to training protocols

    “Training academy” for SP trainers

    SP and SP Trainer Training

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Quality assurance l.jpg

    Quality Assurance

    • Begins with the SP sign-off process

    • Monitoring procedures and analyses:

      • Qualitative (portrayal and scoring accuracy):

        • Live and video review of SP performances

      • Quantitative

        • Score-based analyses

        • Case level and item level comparisons

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Equating procedures l.jpg

    Equating Procedures

    • Within site

      • SP-case combination

    • Between site

      • Central video review

        Data gathering

        Communication and Interpersonal Skills

    • Patient note rater – case combination

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Standard setting l.jpg

    Standard Setting

    • USMLE Standard Setting System

      • Committee assigned responsibility to establish and monitor standards

    • Decision-making process

      • Survey of constituent opinion

      • Standard-setting exercises by independent groups

      • Examinee performance data and score reliability

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Conclusions l.jpg

    Conclusions

    • Numerous logistical and psychometric challenges were identified and responses described

    • In order to ensure exam fairness and consistency:

      • Sound test and case development practices

      • Intensive SP and SP trainer training

      • Rigorous quality assurance

      • Well-considered equating and standard setting procedures

    • Supported by ongoing research on reliability and validity

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


    Speaker contact information l.jpg

    Speaker Contact Information

    Richard E. Hawkins, MD

    National Board of Medical Examiners

    3750 Market Street

    Philadelphia, PA 19104

    215-590-9204 / Fax 215-590-9440

    Email: [email protected]

    Websites:

    http://www.usmle.org

    http://www.nbme.org

    Presented at the 2005 CLEAR Annual Conference

    September 15-17 Phoenix, Arizona


  • Login