Critical Thinking in Medical Education: Assessing What We Mean and What We Know

1. Critical Thinking in Medical Education:Assessing What We Mean and What We Know Ed Krupat, PhD Director, Center for Evaluation Medical Education Grand Rounds Dec. 5, 2008

2. Components of Talks that I Have Liked 1. A bit of context or framing 2. A touch of theoretical/conceptual background 3. A good portion of research and data 4. A bounty of concern for practical application 5. A strong dose of provocative discussion

3. Context: The Origins of These Efforts • Derek Bok--to--Richard Hersh • Evaluation of programs • Confusion-to-clarity-to-confusion-to-??? • Why is critical thinking so important • In life • In medicine • Desire to be data-driven

4. Conceptual Analysis: Just What Is Critical Thinking?? • Overlapping concepts • Analytic reasoning • Problem solving • Decision making • Clinical/diagnostic reasoning/judgement • Habits of mind • Meta-cognition • Adaptive expertise

5. Some Definitions:Critical thinking is… • ...the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication as a guide to belief and action. (Scriven, 1996) • …the art of thinking about your thinking while you are thinking in order to make your thinking better: more clear, more accurate, and more defensible. (Paul et al, 1989)

6. Bloom’s Taxonomy

7. Research & Data • Two projects being conducted simultaneously 1.Qualitative: Do physicians agree about just what critical thinking is? 2.Quantitative: Can we assess critical thinking among medical students?

8. What is Critical Thinking: The Responses of Physicians • Survey of practicing MD faculty at 5 medical schools • HMS, UCSF, Case Western, U Mass, Baylor (total n=73) • Convenience sample

9. Two Tasks • Define critical thinking • Think of a clinical scenario in which critical thinking was important • describe it • state what a good critical thinker would do or say in that situation • state what a poor critical thinker would do or say • state how the outcome would differ if one or the other would have been the physician involved

10. Coding of the Definitions. 1 • Definition specifically included: • Collection of information/data: 27% • Making sense of information: 96% • Utilization for decision making: 74% • Utilization for action: 14% • Specific ties to medical context: 43% • Necessity of building upon knowledge base: 27%

11. Coding of the Definitions. 2 • Critical thinking characterized as • A process • A skill or ability • A disposition

12. Critical Thinking as a Process • …a process of reflective reasoning that uses objective evidence, a deliberate weighing of options and alternatives, and clinical judgement and experience to guide decision making. • …the process by which one is able to rationally acknowledge different choices, processes and outcomes in the clinical encounter. • a process in which problems are being analyzed from different angles and connected to pre-existing knowledge before any conclusions are being drawn 44% of all responses

13. Critical Thinking as a Skill or Ability • …the ability to think through a problem using reasoning. Also the ability to judge the credibility of sources. • …the ability to rigorously weigh the validity of evidence and then to effectively synthesize this evidence to reach a clinical decision. • …the ability to effectively problem solve using known data or under conditions of uncertainty. 50% of all responses

14. Critical Thinking as a Disposition • …careful attention to what you know, vigilance for what you do not, and the courage to question both of the above categories. • …thinking about an topic, issue, or challenge in a way that sets aside my immediate ’gut’ response,so that I can be open and reflective to other possible ways of viewing the challenge… • …thinking deeply, keenly, flexibly, openly, reflectively, with an awareness of self and others, with attention to what is known and unknown, and with humility. 10% of all responses

15. Breakdown by Specialty

16. Clinical Situations: An Interim Report • Looking for • What situations, choices, challenges differentiate critical and non-critical thinkers • What is it that critical thinkers do or say • What is it that non-critical thinkers do or say • What outcomes differ when critical thinking is or is not in evidence • Bottom line: • What are the key differences in the thoughts, words, and actions of critical thinkers

17. Clinical Contexts and Tasks • Not surprising • Diagnosis and treatment • Surprising • Collaboration and interpersonal issues • How to protect patient’s rights and autonomy • How to deal with difficult patient requests • How to provide patient with appropriate options

18. The Differences • Critical thinkers: • Do more data gathering • Avoid premature conclusions • See inconsistencies in information • Utilize knowledge more extensively and explicitly to make decisions • Are aware of limitations and doubts • Monitor and evaluate their own decisions • Involve patients more fully • Provide patients with options

19. A Few Conclusions, Many Questions • Physicians are not necessarily all speaking the same language when they discuss critical thinking with one another • Is critical thinking a skill, a process, or a way of looking at the world? • Are skill and disposition both necessary? • Do you teach/encourage/foster skills in the same way as dispositions??? • Do you assess skills in the same way that you assess dispositions? • Where does building upon a knowledge base come in?

20. How do we assess critical thinking? • Collegiate Learning Assessment (CLA) • Used in over 200 colleges and universities • Assesses higher order outcomes • Students work on 90 minute performance tasks • System has been devised to generate reliable scores • Question: Is this a valid measure of critical thinking for medical students?

21. Catfish: One of the Performance Tasks Used • Grotesquely mutated catfish has been found in the local lake that supplies the town’s water • You will serve on the mayor’s advisory panel • You are provided with 6 documents to read • Newspaper article • Editorial by environmental activist • Radio interview with a biologist from a nearby college • State report on water testing from lake • Area map • Journal article about similar discoveries

22. Task • Open-ended written questions ask students to: • Identify main hypotheses to explain phenomenon • Identify strengths and weakness of each • State and defend most likely explanation • Suggest course of action

23. Scoring • Evaluation of evidence • What is relevant, what is valuable • Analysis and synthesis of evidence • Connections, inconsistencies, flaws in reasoning • Conclusion drawing • Acknowledging alternatives and options • Presentation of arguments • Concise, evidence-based, logically structured

24. Research Design • Recruit multiple schools • Recruit and compare students at two (or more) points in time • Entering Year 1 students took two tasks • Students at end of Year 3 took one task • Look for associations between CLA scores and other performance indicators • MCAT scores • Gender • Age • English as first language • Step scores • Compare scores between first and third year students

25. Analysis Problems • Among year 1 students • scores on the two tasks not as highly correlated as would be expected • Brain Boost task always given first • Year 1 students spend consistently less time and do consistently worse on Catfish (always second task) than on Brain Boost • Among Year 3 students--who only took one task, either Brain Boost or Catfish--students take more time and perform better than on Brain Boost • Small n’s

26. Findings 1. • Medical school students have good critical thinking skills at baseline • Across the 3 schools, year 1’s • fell at the 87th percentile among graduating seniors nationally • were 1.25 standard deviations above the national mean • Is that good news or, with selectivity of the medical schools, should we expect to be even higher??

27. Table 1. CLA performance by task, year, and school ** p<.01

28. Table 2. Correlations among tests and demographic variables Significance codes: * p<.05, ** p<.01, *** p<.001 Note: Significance not adjusted for multiple significance tests

29. Findings to Come • What is the correlation between CLA scores and • Step II CK • Step II CS • HMS Comprehensive Exam

30. What have we learned? • Critical thinking has many dimensions, and physicians may be talking past one another if they don’t share meaning when they speak. • Critical thinking can manifest itself in how to deal with patients, not just in generating diagnoses and treatment plans. • Critical thinking involves being • Sensitive • Inclined • Able (Perkins & Ritchhart 2002)

31. What have we learned? 2 • Medical students enter school with solid critical thinking skills • Critical thinking is somewhat, but not greatly, associated with other measured outcomes • It is still unclear the extent to which medical education fosters or hinders critical thinking • We have more questions at the end of this process than we did than when it began, but the questions are • more sophisticated • more tied to possible action

32. Some Big Questions • If we think that critical thinking is so important, why don’t we explicitly design the curriculum to assure that we “teach” it? • If we wanted to do so, how and when would it be taught? • Should measures of critical thinking, as a skill or disposition, be built into our admissions screening process? • Should critical thinking be a core competency that is assessed throughout the curriculum? as a graduation requirement? As a requirement for licensure?