Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites
This presentation is the property of its rightful owner.
Sponsored Links
1 / 17

Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD, PowerPoint PPT Presentation


  • 45 Views
  • Uploaded on
  • Presentation posted in: General

Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites. Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD, Stephanie F. Donnelly MD, Joan M. VanCamp MD, and Amy L. Waer MD

Download Presentation

Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD,

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Jeffrey g chipman md constance c schmitz phd travis p webb md mohsen shabahang md phd

Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites

Jeffrey G. Chipman MD, Constance C. Schmitz PhD,

Travis P. Webb MD, Mohsen Shabahang MD PhD,

Stephanie F. Donnelly MD, Joan M. VanCamp MD, and

Amy L. Waer MD

University of Minnesota, Department of Surgery

Funded by the Association for Surgical Education

Center for Excellence in Surgical Education, Research & Training (CESERT)

Surgical Education Research Fellowship (SERF)


Introduction

Introduction

  • ACGME Outcome Project

    • Professionalism

    • Interpersonal & Communication skills

  • Need test with validated measures


Professionalism communication

Professionalism & Communication

  • More important than clinical skills in the ICU Crit Care Clin 20:363-80, 2004

    • Communication

    • Accessibility

    • Continuity

  • 1 out of 5 deaths in the US occurs in an ICU Crit Care Med 32(3):638, 2004

  • < 5% of ICU patients can communicate when end-of-life decisions are made

    Am. J. Resp. Crit. Care. Med. 155:15-20, 1997


Family conference osce

Family Conference OSCE

  • Two 20-minute encounters (cases)

    • End-of-life

    • Disclosure of a complication

  • Literature-based rating tools

  • Trained family actors and raters

  • Ratings by family, clinicians, self

  • Debriefing, video

Chipman et al. J Surg Ed, 64(2):79-87, 2007.


Family conference osce minnesota experience

Family Conference OSCEMinnesota Experience

  • High internal consistency reliability

  • Strong inter-rater agreement

  • Raw differences favored PGY3s over PGY1s

  • Small numbers

Chipman et al. J Surg Ed, 64(2):79-87, 2007

Schmitz et al. Crit Care Med 35(12):A122, 2007

Schmitz et al. Simulation in Health Care 3(4):224-238, 2008


Replication study purpose

Replication StudyPurpose

  • Test the feasibility of replicating the OSCE

  • Examine generalizability of scores

    • Institutions

    • Types of raters (clinical, family, resident)

  • Examine construct validity

    • PGY1s vs. PGY3s


Replication study methods

Replication StudyMethods

  • 5 institutions

  • IRB approved at each site

  • Training Conference (Minnesota)

  • Site Training

    • Detailed case scripts

    • Role plays

    • Videos of prior “good” and “bad” performances


Replication study methods learner assessment

Replication StudyMethods – Learner Assessment

  • Assessment by:

    • Clinical raters (MD & RN)

    • Family actors

    • Self

  • Only family raters were blinded

  • Rating forms sent to Minnesota

  • Data analyzed separately for DOC, EOL


Generalizabilty theory

Classical test theory considers only one type of measurement error at a time

Test-retest

Alternate forms

Internal consistency

Inter-rater agreement

Generalizability theory allows for errors that occur from multiple sources

Institutions

Rater type

Family actors

Provides overall summary as well as breakdown by error sources and their combinations

Generalizabilty Theory

Mushquash C & O’Connor. SPSS and SAS programs for generalizability theory analyses Behavior Research Methods 38(3):542-47, 2006


Generalizabilty theory1

Generalizabilty Theory

  • Summary statistics (0 to 1)

    • 1.0 = perfectly reliable (generalizable) assessment

  • Relative generalizability

    • Stablility in relative position (rank order)

  • Absolute generalizability

    • Agreement in actual score


Results feasibility

ResultsFeasibility

  • N = 61 residents

  • Implementation fidelity was achieved at each site

  • Key factors:

    • Local surgeon champions

    • Experienced standardized patient program

    • On-site training (4 hrs) by PIs

    • Standardized materials & processes


Results internal consistency reliability

ResultsInternal Consistency Reliability


Results generalizability

ResultsGeneralizability

The relative G-coefficients we obtained suggest the exam results can be used for formative or summative classroom assessment.

The absolute G-coefficients suggest we wouldn’t want to set a passing score for the exam.

Downing. Reliability: On the reproducibility of assessment data Med Educ 38:19009-1010, 2004


Results construct validity manova

ResultsConstruct ValidityMANOVA

Disclosure

End-of-Life

p = 0.44

p = 0.41

Between subjects effect (PGY 1 vs. PGY 3) was not significant (p = 0.66 DOC, p = .0.26 EOL).


Study qualifications

Study Qualifications

  • Only family members were blinded

    • Clinician and family ratings were significantly correlated on EOL & DOC

  • Nested vs. fully crossed design


Conclusions family conference osce

ConclusionsFamily Conference OSCE

  • Feasible at multiple sites

  • Generalizeable Scores

    • Useful for formative, summative feedback

    • Raters were greatest source of error variance

  • Did not demonstrate construct validity

    • Questions the assumption that PGY-3 residents are inherently better than PGY-1 residents, particularly in communication


Study partners lurcat group

Study PartnersLurcat Group

  • Amy Waer, MD

    • University of Arizona

  • Travis Webb, MD

    • Medical College of Wisconsin

  • Joan Van Camp, MD

    • Hennepin County Medical Center

  • Mohsen Shabahang, MD, PhD

    • Scott & White Clinic, Texas A&M

  • Stephanie Donnelly MD

    • Mayo Clinic Rochester

  • Connie Schmitz, PhD

    • University of Minnesota

Acknowledgments

Jane Miller, PhD, and Ann Wohl, University of Minnesota IERC (Inter-professional Education Resource Center)

Michael G. Luxenberg, PhD, and Matt Christenson, Professional Data Analysts, Inc., Minneapolis, Minnesota


  • Login