Quality & Quantity of Teaching & Learning Opportunities

1. Reactions of Surgical Program Directors to a Web-Based Interactive Educational Program Focusing on Cognitive Skills Donald A. Risucci, PhD1, Patricia J. Numann, MD2, Richard Welling, MD3, Marsha F. Pfingsten, MEd4, Patrice Gabler Blair, MPH4, Ajit K. Sachdeva, MD4 1New York Medical College, Valhalla NY 2SUNY Upstate Medical University, Syracuse, NY 3Good Samaritan Hospital, Cincinnati, OH 4American College of Surgeons, Chicago, IL

2. Quality & Quantity of Teaching & Learning Opportunities Traditional Non-Traditional

3. Quality & Quantity of Teaching & Learning Opportunities Traditional Skills Labs Work hour restrictions Non-Traditional

4. Quality & Quantity of Teaching & Learning Opportunities Traditional Simulation Skills Labs Work hour restrictions Patient Safety Concerns Non-Traditional

5. Quality & Quantity of Teaching & Learning Opportunities Traditional Computer-Based Learning Simulation Skills Labs Work hour restrictions Patient Safety Concerns Diminished Reimbursement Non-Traditional

6. Quality & Quantity of Teaching & Learning Opportunities Traditional Immersive technologies Computer-Based Learning Simulation Skills Labs Work hour restrictions Patient Safety Concerns Diminished Reimbursement Increased Specialization Non-Traditional

7. Quality & Quantity of Teaching & Learning Opportunities Traditional Web 2.0, 3.0….. Immersive technologies Computer-Based Learning Simulation Skills Labs Work hour restrictions Patient Safety Concerns Diminished Reimbursement Increased Specialization Advanced Technologies Non-Traditional

8. Seasonal Variation in Surgical Outcomes as Measured by the American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) Ann Surg. 2007 September; 246(3): 456–465. Michael J. Englesbe, MD,* Shawn J. Pelletier, MD,* John C. Magee, MD,* Paul Gauger, MD,* Tracy Schifftner, MS,† William G. Henderson, PhD,† Shukri F. Khuri, MD,‡ and Darrell A. Campbell, MD* From the *Department of Surgery, University of Michigan Health System, Ann Arbor, MI; †Department of Veterans Affairs, VA—National Surgical Quality Improvement Program, Aurora, CO; and ‡National Surgical Quality Improvement Program, West Roxbury, MA.

11. Fundamentals of Surgery Curriculum American College of Surgeons Division of Education www.facs.org/education/fundamentalsofsurgery.html

12. Focus on Diagnosis Diagnosis errors far outnumber medication errors as a cause of claims lodged (26% v. 12% in one study; Sato, 2001); 32% v. 8% in another study (Phillips et al 2004). Harris poll commissioned by National Patient Safety Foundation: one in six people had personally experienced a medical error related to misdiagnosis (Golodner 2004). Most medical error studies find that 10–30 percent (range = 0.6–56.8 percent) of errors are errors in diagnosis. A recent review of 53 autopsy studies found an average rate of 23.5% major missed diagnoses (range = 4.1–49.8 percent). Disease-specific studies show that substantial percentage of patients (range = 2.1% – 61%) experienced missed or delayed diagnoses. Schiff et al, 2005

13. Fundamentals of Surgery Curriculum • Case-based interactive curriculum • Delivered on-line to PGY-1 surgical residents • Each scenario presents a patient; resident can “take a history”, “perform” a physical examination, order tests, review records, and recommend therapeutic interventions. • Toolbar allows access to the patient’s virtual chart, including appropriate reports, and archives the information gathered during the patient history, physical examination, and diagnostic testing.

14. Fundamentals of Surgery Curriculum • Specific feedback for every action taken while interviewing, examining, diagnosing and managing the patient. • If a resident’s actions in caring for a virtual patient result in dire consequences, the resident is required to start over. • Upon completion of FSC, the resident will have diagnosed and recommended treatment for scores of “patients

15. Fundamentals of Surgery Curriculum • Learning exercise, not a summative assessment tool: residents can explore various alternatives and receive feedback without concern about being formally evaluated. • When complete, FSC will be composed of 11 on-line modules with approximately 100 interactive case-based scenarios representing the foundation of surgical cognitive content for PGY-1 residents across all surgical specialties.

16. A Scenario may open with a Learning Objective Page, indicated by a Surgeon behind a Podium.

17. This is a Reference Page, indicated by a Surgeon with an Open Book.

18. Scenario Authors provide a detailed description of the patient. Images are created to exhibit signs that the Resident is expected to notice and to incorporate into the diagnostic process..

19. Question Possible Choices Progress Meter key When the Resident opens a work space, a list of choices is presented, a key to the “grade” that each choice can receive, and a Progress Meter..

20. When a selection is made, in this case a question to be posed to the patient, the response appears as illustrated above.

21. In this case, a critical question was selected, as indicated by the green dot. The progress meter is partially full, indicating that additional critical choices must be made before the Resident can continue. “grade” Progress Meter

22. After all critical choices have been made, the Progress Meter registers 100%, The Resident is allowed to Continue to the next step in evaluating the patient. After all correct responses have been selected

23. The Resident can review the patient’s responses to the history by selecting the Patient History and Image Tab from the Patient Section of The Toolbar. The Toolbar represents the patient’s Virtual Chart.

24. The Resident can review the patient’s Vital signs by selecting the Vital Signs Tab from the Patient Section of The Toolbar.

25. Reports and test results which are in the patient’s chart at the time of the encounter are made available in the Records Section and the Tests Section of the Toolbar at the start of the branch. If a report or test is ordered within a branch, it is made available at the appropriate time.

26. Each selection receives a response that not only indicates whether it is “right” or “wrong”, but also provides feedback designed to enhance the educational experience for the Resident.

27. After the Resident has successfully completed the scenario, a Summary is presented. If the Learning Objective was not presented at the beginning of the scenario, it is presented here.

28. Key Learning Points that have been demonstrated within the scenario are listed after the Summary, to reinforce the learning

29. A Resident may choose to review the Scenario at this time, or to return to the Menu for another scenarios. All scenarios, as well as all reference material, tables, and charts, remain available to each Resident for the entire year. Feedback The Fundamentals of Surgery Curriculum is an educational program, not a test. For that reason, and to encourage each Resident to investigate each scenario completely, personal feedback is provided to the Resident; however this is not tracked in any way.

30. Current Study Evaluates Program Directors’ reactions to a preview of a scenario from the Fundamentals of Surgery Curriculum.

31. Methods Program Directors were invited via the Listserv to preview the Fundamentals of Surgery Curriculum during the October 2007 ACS Clinical Congress. Previews were scheduled for 31 Program Directors on a first-come first-serve basis. Each Program Director spent ~ 1 hour engaged in the scenario

32. Methods After engaging in a scenario each Program Director completed a questionnaire requesting: • Age range • Rating of comfort using computers • Ratings of the scenario’s utility in addressing 9 broadly defined educational goals (e.g., Provides a Solid Foundation for Future Learning) • Ratings of 6 separate features of the scenario (e.g., Ease of Use, Feasibility). Informal debriefings were also conducted and comments were collected.

35. Program Directors with higher levels of comfort using computers tended to perceive the modules as easier to use (p < 0.01)

36. Ratings of Features Ease of use (logical flow and methods of navigation) Very difficult to use Extremely easy to use Usefulness of multimedia components to present material and enhance understanding Does not enhance presentation Extremely useful Quality of presentation (appealing color schemes, format, and graphics) Very Poor Exceptional

37. Ratings of Features Extent to which interactivity within the scenario promotes engagement in solving the problem presented Not at all engagingMade me want to solve the problem Extent to which immersion in this virtual experience adequately recreates the environment in which the scenario takes place Felt very artificial Felt real Feasibility of incorporating scenarios like this one into the required PGY-1 curriculum Will not be feasible Should be part of required curriculum

38. Ratings of FSC Utility in Addressing Educational Goals • Prepares residents to apply critical thinking and judgment essential in clinical situations during the PGY-1 year. • Improves residents’ patient care skills and confidence during the PGY-1 year. • Delivers content consistent with current practices and/or evidence (when available).

39. Ratings of FSC Utility in Addressing Educational Goals • Provides a solid foundation for future learning. • Challenges residents. • Encourages self-assessment and reflection. • Motivates residents to voluntarily complete other scenarios. • Improves residents’ teamwork and professionalism. • Increases resident satisfaction with the educational process.

40. Ratings of the Scenario’s Utility for Addressing 9 Broadly Defined Educational Goals

41. Comments: Informal Debriefing • Scenarios were fun and interesting. • Felt as if they were diagnosing a real patient; very involved in scenario. • Several asked about the availability of other modules. • Enables delivery of useful information consistently and efficiently. • Helps to address educational objectives within the 80 hour work. • Exposes residents to cases they may not actually encounter.

42. Comments: Informal Debriefing • Difficulties with the original physical examination model were recorded and have led to enhancements in that component of the scenarios. • Most Program Directors indicated interest in participating in future content reviews, beta testing, and research initiatives. • Some Program Directors indicated that they would like to require residents to complete FSC prior to entering their program.

43. Evaluation Framework • Reactions: Perceptions of stakeholders & users • Learning: Cognitive gains from engaging in the curriculum • Behavior: Changes in clinical performance resulting from exposure to curriculum • Outcomes: Effects on patient outcomes, quality improvement

44. Conclusions Program Directors generally perceived the Fundamentals of Surgery Curriculum as a very feasible and effective tool addressing important educational needs of entering PGY-1 residents. Positive reactions from Program Directors suggest that the Fundamentals of Surgery Curriculum are likely to be implemented and evaluated in a significant number of programs.