slide1
Download
Skip this Video
Download Presentation
Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD,

Loading in 2 Seconds...

play fullscreen
1 / 17

Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD, - PowerPoint PPT Presentation


  • 81 Views
  • Uploaded on

Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites. Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD, Stephanie F. Donnelly MD, Joan M. VanCamp MD, and Amy L. Waer MD

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD, ' - masato


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites

Jeffrey G. Chipman MD, Constance C. Schmitz PhD,

Travis P. Webb MD, Mohsen Shabahang MD PhD,

Stephanie F. Donnelly MD, Joan M. VanCamp MD, and

Amy L. Waer MD

University of Minnesota, Department of Surgery

Funded by the Association for Surgical Education

Center for Excellence in Surgical Education, Research & Training (CESERT)

Surgical Education Research Fellowship (SERF)

introduction
Introduction
  • ACGME Outcome Project
    • Professionalism
    • Interpersonal & Communication skills
  • Need test with validated measures
professionalism communication
Professionalism & Communication
  • More important than clinical skills in the ICU Crit Care Clin 20:363-80, 2004
    • Communication
    • Accessibility
    • Continuity
  • 1 out of 5 deaths in the US occurs in an ICU Crit Care Med 32(3):638, 2004
  • < 5% of ICU patients can communicate when end-of-life decisions are made

Am. J. Resp. Crit. Care. Med. 155:15-20, 1997

family conference osce
Family Conference OSCE
  • Two 20-minute encounters (cases)
    • End-of-life
    • Disclosure of a complication
  • Literature-based rating tools
  • Trained family actors and raters
  • Ratings by family, clinicians, self
  • Debriefing, video

Chipman et al. J Surg Ed, 64(2):79-87, 2007.

family conference osce minnesota experience
Family Conference OSCEMinnesota Experience
  • High internal consistency reliability
  • Strong inter-rater agreement
  • Raw differences favored PGY3s over PGY1s
  • Small numbers

Chipman et al. J Surg Ed, 64(2):79-87, 2007

Schmitz et al. Crit Care Med 35(12):A122, 2007

Schmitz et al. Simulation in Health Care 3(4):224-238, 2008

replication study purpose
Replication StudyPurpose
  • Test the feasibility of replicating the OSCE
  • Examine generalizability of scores
    • Institutions
    • Types of raters (clinical, family, resident)
  • Examine construct validity
    • PGY1s vs. PGY3s
replication study methods
Replication StudyMethods
  • 5 institutions
  • IRB approved at each site
  • Training Conference (Minnesota)
  • Site Training
    • Detailed case scripts
    • Role plays
    • Videos of prior “good” and “bad” performances
replication study methods learner assessment
Replication StudyMethods – Learner Assessment
  • Assessment by:
    • Clinical raters (MD & RN)
    • Family actors
    • Self
  • Only family raters were blinded
  • Rating forms sent to Minnesota
  • Data analyzed separately for DOC, EOL
generalizabilty theory
Classical test theory considers only one type of measurement error at a time

Test-retest

Alternate forms

Internal consistency

Inter-rater agreement

Generalizability theory allows for errors that occur from multiple sources

Institutions

Rater type

Family actors

Provides overall summary as well as breakdown by error sources and their combinations

Generalizabilty Theory

Mushquash C & O’Connor. SPSS and SAS programs for generalizability theory analyses Behavior Research Methods 38(3):542-47, 2006

generalizabilty theory1
Generalizabilty Theory
  • Summary statistics (0 to 1)
    • 1.0 = perfectly reliable (generalizable) assessment
  • Relative generalizability
    • Stablility in relative position (rank order)
  • Absolute generalizability
    • Agreement in actual score
results feasibility
ResultsFeasibility
  • N = 61 residents
  • Implementation fidelity was achieved at each site
  • Key factors:
    • Local surgeon champions
    • Experienced standardized patient program
    • On-site training (4 hrs) by PIs
    • Standardized materials & processes
results generalizability
ResultsGeneralizability

The relative G-coefficients we obtained suggest the exam results can be used for formative or summative classroom assessment.

The absolute G-coefficients suggest we wouldn’t want to set a passing score for the exam.

Downing. Reliability: On the reproducibility of assessment data Med Educ 38:19009-1010, 2004

results construct validity manova
ResultsConstruct Validity MANOVA

Disclosure

End-of-Life

p = 0.44

p = 0.41

Between subjects effect (PGY 1 vs. PGY 3) was not significant (p = 0.66 DOC, p = .0.26 EOL).

study qualifications
Study Qualifications
  • Only family members were blinded
    • Clinician and family ratings were significantly correlated on EOL & DOC
  • Nested vs. fully crossed design
conclusions family conference osce
ConclusionsFamily Conference OSCE
  • Feasible at multiple sites
  • Generalizeable Scores
    • Useful for formative, summative feedback
    • Raters were greatest source of error variance
  • Did not demonstrate construct validity
    • Questions the assumption that PGY-3 residents are inherently better than PGY-1 residents, particularly in communication
study partners lurcat group
Study PartnersLurcat Group
  • Amy Waer, MD
    • University of Arizona
  • Travis Webb, MD
    • Medical College of Wisconsin
  • Joan Van Camp, MD
    • Hennepin County Medical Center
  • Mohsen Shabahang, MD, PhD
    • Scott & White Clinic, Texas A&M
  • Stephanie Donnelly MD
    • Mayo Clinic Rochester
  • Connie Schmitz, PhD
    • University of Minnesota

Acknowledgments

Jane Miller, PhD, and Ann Wohl, University of Minnesota IERC (Inter-professional Education Resource Center)

Michael G. Luxenberg, PhD, and Matt Christenson, Professional Data Analysts, Inc., Minneapolis, Minnesota

ad