Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites. Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD, Stephanie F. Donnelly MD, Joan M. VanCamp MD, and Amy L. Waer MD
Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites
Jeffrey G. Chipman MD, Constance C. Schmitz PhD,
Travis P. Webb MD, Mohsen Shabahang MD PhD,
Stephanie F. Donnelly MD, Joan M. VanCamp MD, and
Amy L. Waer MD
University of Minnesota, Department of Surgery
Funded by the Association for Surgical Education
Center for Excellence in Surgical Education, Research & Training (CESERT)
Surgical Education Research Fellowship (SERF)
Am. J. Resp. Crit. Care. Med. 155:15-20, 1997
Chipman et al. J Surg Ed, 64(2):79-87, 2007.
Chipman et al. J Surg Ed, 64(2):79-87, 2007
Schmitz et al. Crit Care Med 35(12):A122, 2007
Schmitz et al. Simulation in Health Care 3(4):224-238, 2008
Classical test theory considers only one type of measurement error at a time
Generalizability theory allows for errors that occur from multiple sources
Provides overall summary as well as breakdown by error sources and their combinations
Mushquash C & O’Connor. SPSS and SAS programs for generalizability theory analyses Behavior Research Methods 38(3):542-47, 2006
The relative G-coefficients we obtained suggest the exam results can be used for formative or summative classroom assessment.
The absolute G-coefficients suggest we wouldn’t want to set a passing score for the exam.
Downing. Reliability: On the reproducibility of assessment data Med Educ 38:19009-1010, 2004
p = 0.44
p = 0.41
Between subjects effect (PGY 1 vs. PGY 3) was not significant (p = 0.66 DOC, p = .0.26 EOL).
Jane Miller, PhD, and Ann Wohl, University of Minnesota IERC (Inter-professional Education Resource Center)
Michael G. Luxenberg, PhD, and Matt Christenson, Professional Data Analysts, Inc., Minneapolis, Minnesota