1 / 50

Assessing operative autonomy Combining theory and software to make evaluation easy

Assessing operative autonomy Combining theory and software to make evaluation easy. Jonathan Fryer MD, Professor of Surgery, Feinberg School of Medicine, Northwestern University. Disclosures. I have made no financial gains from this project I may in the future

mili
Download Presentation

Assessing operative autonomy Combining theory and software to make evaluation easy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Assessing operative autonomy Combining theory and software to make evaluation easy Jonathan Fryer MD, Professor of Surgery, Feinberg School of Medicine, Northwestern University

  2. Disclosures • I have made no financial gains from this project • I may in the future • I intend to continue work on this project regardless

  3. What is the most essential goal of surgical training?

  4. Operative Autonomy • The ability to independently perform operations safely and effectively.

  5. The Problem There is growing concern that graduating surgical residents are not achieving operative autonomy with essential procedures. Bell RH. Why Johnny cannot operate. Surgery146, 533–542 (2009). Mattar SGet al. General Surgery Residency Inadequately Prepares Trainees for Fellowship: Results of a Survey of Fellowship Program Directors. Annals of Surgery September 2013258, 440–449 (2013). Coleman JJ et al. Early Subspecialization and Perceived Competence in Surgical Training: Are Residents Ready? Journal of the American College of Surgeons216, 764–771 (2013). Chen P. Are Today’s New Surgeons Unprepared? Well (2013). at http://well.blogs.nytimes.com/2013/12/12/are-todays-new-surgeons-unprepared

  6. The Problem • To be able to fix it…… You have to be able to measure it.

  7. The Problem • We don’t do a very good job of assessing residents in the OR.

  8. The Problem • Currently, summative assessment of OR performance is based on: • # of cases logged by resident • Role of resident in each case? • Semi-annual global evaluations • Memory decay?

  9. The Problem …asking busy surgical faculty to fill out complex assessment forms in a timelymanner, doesn’t work.

  10. The Solution A simple assessment tool that: • Assesses operative autonomy • Doesn’t impede surgical workflow • Facilitates high compliance and prompt completion

  11. Theoretical Framework • Inter-related constructs: • Supervision, Guidance, Autonomy, Performance • Faculty Supervision (oversight) ≠ • Faculty guidance (physical or verbal help) • 1 • Faculty Guidance = Resident Autonomy • Resident Autonomy = ƒ (Resident performance)

  12. The Solution • With every case faculty: • Provide resident supervision. • Assess and document the level of operative autonomy achieved by the resident. • Progressively reduce the level of operative guidance they provide to resident.

  13. The “Zwisch” Scale • 4 levels of operative guidance • Show & Tell • Active Help • Passive Help • Supervision Only DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing Residents in the Operating Room. Journal of Surgical Education70, 24–30 (2013).

  14. Our method: PASS (Procedural Autonomy and Supervision System)

  15. Today

  16. Coming soon…

  17. Study Design: Participants and Setting • Department of general surgery at a large academic hospital • All teaching faculty underwent formal frame-of-reference training per published protocol1 • All general surgery residents and trained faculty raters eligible for inclusion • IRB-approved 1George et al, J. Surg. Educ.2013; 70

  18. Results: Feasibility • A 1 hour rater training session is sufficient to achieve reliable and accurate ratings1 • 92% response rate using PASS 1George, B. C. et al. Duration of Faculty Training Needed to Ensure Reliable OR Performance Ratings. J. Surg. Educ. 70, 703–708 (2013).

  19. Results: PASS Sample (7 mos)

  20. Results: PASS Sample

  21. Results: Validity: Zwisch Levels by PGY p=<.001 p=<.001 p=<.001 p=0.21 p-values for sequential pair-wise distributions 23.2%

  22. Results: Validity: Zwisch Levels by Complexity p-values for sequential pair-wise distributions p=<.001 p=<.001

  23. Results: Validity: Zwisch Level by Prior Experience p-values for sequential pair-wise distributions p=<.001

  24. Study Design: Data Collection • Sample 2: Video Sample • 8 procedures video recorded for additional review (subset of PASS sample) • Rated by operating faculty, in-person OR observer, and video reviewer using Zwisch scale (blinded to other scores) • Rated by 2 additional video reviewers using other OR assessment instruments (modified OPRS and O-SCORE)

  25. Results: Video Sample

  26. Results: Reliability • Inter-rater reliability • Zwisch ratings • Operating attending, OR observer, and video rater • ICC = .90, 95% CI = .72 - .98, p < .001.

  27. Results: Validity: Zwisch Level correlation with other OR assessment tools

  28. Benefits • Faculty and residents constantly reminded of ultimate goal …. i.e. operative autonomy. • Establishes a conceptual framework for teaching and learning in the OR. • Data can be used to: • Help faculty and residents to set learning goals. • Help programs monitor operative progress and identify those who may need additional attention. • Address regulatory requirements for OR supervision and operative performance assessment. • Establish national norms

  29. Limitations • So far, studied only at a single institution • Validity analysis based on small convenience sample • Raters not blinded to resident PGY level • Comparison with only selected items of OPRS and O-SCORE • Unmeasured confounders (time of day, supervising surgeon experience, etc)

  30. Conclusion • The Zwisch rating scale is a reliable and valid measure of faculty guidance and resident autonomy • Deployed on PASS the Zwisch scale can be used to feasibly record evaluations for the vast majority of operations performed by residents

  31. Vision • All surgical subspecialties. • Other procedural specialties. • Other medical professionals who need to learn to perform complex clinical tasks. • Other trades or professions where trainees need to learn to independently perform complex tasks safely and effectively.

  32. Acknowledgements Surgical Education Research & Development Team Jonathan Fryer Shari Meyerson Debra DaRosa Eric Hungness Research supported by: Excellence in Academic Medicine Program from the State of Illinois Augusta Webster Educational Innovation Grant from the Northwestern University Center for Education in Medicine Mary Schuller Ezra Teitelbaum Brian George Jay Zwischenberger

  33. Theoretical basis Regehr, G., MacRae, H., Reznick, R. K. & Szalay, D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73, 993–997 (1998). Chen, X. (Phoenix), Williams, R. G., Sanfey, H. A. & Dunnington, G. L. How do supervising surgeons evaluate guidance provided in the operating room? The American Journal of Surgery203, 44–48 (2012). George, B., Teitelbaum, E., DaRosa, D., Hungness, E., Meyerson, S., Fryer, J., Schuller, M., Zwischenberger, J. Duration of Faculty Training Needed to Ensure Reliable O.R. Performance Ratings. Journal of Surgical Education 70(6), 703-708 (2013). Global assessment of performance is simpler, more accurate, and more reliable than checklists1 Faculty guidance is related to resident performance2 Faculty can accurately and reliably rate the amount of guidance provided to residents3

  34. Study • Over 7 months • 1490 evaluations • 27 faculty • 31 residents

  35. Study Design: Rating Scales • Zwisch • Procedural Complexity • Operative Performance Rating System (OPRS)1 • 6 general items only--excludes items that pertain only to specific procedures • Ottawa Surgical Competency Operating Room Evaluation (O-SCORE)2 • 5 intra-operative items only--excludes items that did not pertain to intra-operative performance. 1Chen et al, The American Journal of Surgery 2012; 203 2Gofton et al, Acad. Med. 2012; 87

  36. Results: Validity • Convergent Validity for Guidance/Autonomy and Resident Performance • Zwisch level vs. PGY • Zwisch level vs. Complexity • Zwisch level vs. Resident Experience • Construct Validity for Guidance/Autonomy • Zwisch level vs. OPRS guidance item • Construct Validity for Resident Performance • Zwisch level vs. OPRS performance items • Zwisch level vs. O-SCORE performance items

  37. The Team • Dr. Debra DaRosa • Dr. Brian George • Dr. Shari Meyerson • Dr. Ezra Teitelbaum • Mary Schuller • Dr. Nathaniel Soper • Dr. Joseph Zwischenberger

  38. Impact so far • Over 1000 evaluations collected in 6 months • Response rate > 90% • Changes in teaching • They love to use it!

  39. Next steps • Dictation of feedback • Reports

  40. Results: Validity • Convergent Validity for Guidance/Autonomy and Resident Performance • Zwisch level vs. PGY • Zwisch level vs. Complexity • Zwisch level vs. Resident Experience • Construct Validity for Guidance/Autonomy • Zwisch level vs. OPRS guidance item • Construct Validity for Resident Performance • Zwisch level vs. OPRS performance items • Zwisch level vs. O-SCORE performance items

  41. Theoretical Framework Watching Helping

  42. Next Steps • I am actively trying to bring this to MGH • It needs additional development before it can be launched here • Multiple other departments have already committed to supporting this project

  43. Questions?

  44. Results

  45. Results 50% = 60 procedures

  46. Current Status

  47. Road Map

  48. 12 month budget

  49. The “Zwisch” Scale DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing Residents in the Operating Room. Journal of Surgical Education70, 24–30 (2013). • 4 levels of guidance • Show & Tell • Active Help • Passive Help • Supervision Only

  50. Theoretical Framework • Stritter FT et al., Handbook for the academic physician. 1986. • Chen et al., The American Journal of Surgery2012; 203 • Gofton et al., Acad. Med. 2012; 87

More Related