1 / 24

Brent Zabolotny 1 , Robert Gagnon 2 , Bernard Charlin 2 and Sarkis Meterissian 1

Is the Script-Concordance Test a Valid Instrument for Assessment of Intra-operative Decision-making Skills?. Brent Zabolotny 1 , Robert Gagnon 2 , Bernard Charlin 2 and Sarkis Meterissian 1 1 Division of General Surgery and Center for Medical Education, McGill University

trung
Download Presentation

Brent Zabolotny 1 , Robert Gagnon 2 , Bernard Charlin 2 and Sarkis Meterissian 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Is the Script-Concordance Test a Valid Instrument for Assessment of Intra-operative Decision-making Skills? Brent Zabolotny1, Robert Gagnon2, Bernard Charlin2 and Sarkis Meterissian1 1Division of General Surgery and Center for Medical Education, McGill University 2Unit for Research and Development in Health Sciences Education, University of Montreal

  2. The Basics of Medical Reasoning • Medical reasoning is a hypothetico-deductive process (Ellstein 1978, Barrows et al 1982) involving: • Early generation of hypotheses • Oriented data collection • Decision-making judgements

  3. Problem-solving in Clinical Medicine Well-defined problems are rare: Clear data, clear goals, clear solutions Technical knowledge Ill-defined problems are common: Uncertain data, unclear goals, different solutions Professional knowledge Donald A Schön, The Reflective Practitioner, 1983

  4. What is an Expert? • Experts can store, use and retrieve knowledge more efficiently than novices • Experts possess elaborated networks of knowledge fitted to the tasks they regularly do These networks are called SCRIPTS (Feltovich 1983, Schmidt 1990, Charlin 2000)

  5. SCRIPT Theory • In clinical situations, clinicians fill their working memory with knowledge related to each relevant hypothesis • Then a deductive process is used to seek information to either accept or refute each respective hypothesis • A test has been developed that measures this deductive-reasoning process – The Script Concordance Test (SCT)

  6. Script Concordance Test • A challenging context • Authentic clinical situation • Not enough data or conflict - Relevant options • A Likert scale • Capture opinion • Scoring system • Takes into account variability of experts • Aggregate scoring method

  7. The Qualities of a Good SCT • There must be at least 50-60 items to achieve a reliability coefficient (Cronbach alpha) of 80 (r = .80) • Questions should not lead to a single answer (they must not be clearcut) otherwise the SCT would really be an MCQ • Conversely, the answers should not be too widely spread thus indicating a vague question

  8. Evidence that this works?

  9. Purpose To develop and validate a script-concordance test for assessment of clinical reasoning (ie: intra-operative decision-making) in General Surgery Residents.

  10. Methods: SCT Format

  11. Methods: Example (1)

  12. Methods:Example (2)

  13. Methods: Scoring an SCT • The scoring grid of an SCT should be derived by administering it to 10 experts (aggregate scoring method) • Based on their responses some questions (if too vague or too clearcut) can be eliminated • Scoring grid takes into account variability due to the uncertainty of the clinical situation

  14. The Scoring Grid

  15. Methods • Participants: 36 out of the 40 McGill surgical residents wrote the exam • Panel of experts: 10 board-certified general surgeons

  16. Methods Statistical Analysis: • Reliability: Cronbach alpha coefficient • Analysis of item-to-total correlation used to select the best items for the final analysis • Construct validity tested with a one-way ANOVA with post-hoc comparisons test and planned contrasts • All p values at alpha<5% were considered significant

  17. Results • Developed 196 questions • Face and content validity assessed by 3 independent Board-certified General Surgeons • Final exam consisted of the best 100 questions

  18. Results • Initial test: 100 items Cronbach alpha : 0.70 • Optimisation of the items (elimination of items with an inter-item correlation less than 10%) : final test used for analysis = 62 items • Cronbach alpha of the optimised test: 0.85

  19. Resident Level n Mean Score (%) Std Deviation (%) R1 8 52.5 9.96 R2 6 62.5 5.12 R3 9 68.3 9.19 R4 6 75.7 9.61 R5 7 68.0 6.44 Results Examination Scores (62-item test) by Resident Level

  20. Results

  21. Discussion • The SCT developed in this study demonstrated: 1. excellent internal reliability 2. Good construct validity

  22. Discussion: Questions: • Correlation with in-training evaluations? • Correlation with oral exam scores? • Ideal number of panel members? • Makeup of the panel members? subspecialists vs general surgeons • Can the SCT identify residents having difficulty with clinical reasoning?

  23. More on • Web site: http://www.cme.umontreal.ca/tcs • E-mail : Sarkis.meterissian@muhc.mcgill.ca bernard.charlin@umontreal.ca

  24. Results • Evaluation of the exam: - 64% found the SCT tested their intra-operative decision-making skills - 50% found the exam initially difficult but 79% found that the exam became easier as they better understood the format - 71% found that the SCT exam was a valuable and educational examination tool

More Related