Question Writing Workshop Stanford University Medical Education Faculty

1. Question Writing WorkshopStanford University Medical Education Faculty Lynn C. Webb, Ed.D. August 15, 2007 testing@lwebb.com

2. Agenda • Introductions • Examination goals • Giving away the answers • 3 formats for questions • Investing in future examinations • The joy of feedback • Questions / discussion

3. 1. Introductions • Lynn Webb – Testing consultant • Previous affiliations • American Board of Psychiatry & Neurology • National Board of Medical Examiners • ETS (Princeton) • ACT (Iowa City)

4. AAN’s RITE ACP’s PRITE APA’s FOCUS AOA’s UHM ABPM AAPM&R’s SAE-R WIP CBNC ABPN Certification Recertification Subspecialty Added qualifications Non-medical exams NCLEX Solar Energy Examples of Tests

5. 2. Examination Goals • Test students’ learning • Validate your teaching • Prepare students for future testing Your questions can contribute to these goals, or detract from them.

6. Contribute to Goals • Questions should be aligned with the specific content • Questions should be technically sound • Clearly written • Straight-forward, NOT tricky

7. Detract from Goals • Questions don’t address teaching or learning • Questions are convoluted or confusing, testing something other than the content • The focus shifts to the question instead of the content

8. 3. Giving away the answers • Medical students tend to be test-wise • When they don’t know an answer, they retrace your steps in writing the question to guess the correct answer • Even if YOU are test-wise, blind-spots exist when it’s your material • Multiple-choice questions are especially vulnerable

9. Let’s take a quiz • Our content will be highly esoteric facts about the Alaskan Malamute • Each quiz question will contain a flaw in its structure that will assist test-wise examinees. You won’t need to study the Alaskan Malamute to pass this quiz.

10. Boris Badenov Webb

11. 1. The primary function of sled dogs such as the Alaskan Malamute, Samoyed, Siberian Husky, and Eskimo Dog is to: • rescue drowning humans • retrieve tossed objects • guard property • pull sleds

12. Test-wise examinees • Look for hints that the question writer overlooked between the question and the correct option • Sled dogs – Pull sleds (no other options about sleds)

13. 2. Sled dogs provided vital transport of a life-saving serum during a diphtheria epidemic in Nome, Alaska in: • 1492 • 1776 • 1925 • 1952

14. Several ways to approach question • First, eliminate options that aren’t plausible: 1492 and 1776 are too early for question content • Now you have 50/50 chance between C and D. • Is D plausible? Why not fly in 1952? • Numeric options – when in doubt, choose C

15. 3. Credit for the introduction of the Alaskan Malamute in the ‘lower 48’ is usually given to: • Arthur Treadwell Walden and The Seeleys • Admiral Byrd and Arthur Treadwell Walden • The Seeleys and Commodore Perry • Arthur Treadwell Walden and Jack London

16. Look for repeats • Arthur Treadwell Walden and the Seeleys • Admiral Byrd and Arthur Treadwell Walden • The Seeleys and Commodore Perry • Arthur Treadwell Walden and Jack London

17. More repeats? • Arthur Treadwell Walden and The Seeleys • Admiral Byrd and Arthur Treadwell Walden • The Seeleys and Commodore Perry • Arthur Treadwell Walden and Jack London

18. Some guys listed only once • Arthur Treadwell Walden and The Seeleys • Admiral Byrd and Arthur Treadwell Walden • The Seeleys and Commodore Perry • Arthur Treadwell Walden and Jack London

19. Where do repeats converge? • Arthur Treadwell Walden and The Seeleys • Admiral Byrd and Arthur Treadwell Walden • The Seeleys and Commodore Perry • Arthur Treadwell Walden and Jack London

20. Why do test-wise examinees do that? • They are retracing your steps in creating the question • That’s why you shouldn’t submit your first draft • It’s easy to fix this problem

21. 4. Champion Coldfoot Oonanik, the world’s most titled dog, was how many inches at the withers? • 2.75” • 25.0” • 27.5” • 275”

22. Test-wise examinees compare options • 3 options have the digits 2 – 7 – 5 • One option has different digits (B), but it’s close in size to C • C must be the answer • (Author had the answer 27.5, then moved the decimal place for 2 distractors, then chose another plausible one near answer)

23. Alaskan Malamutes are known for a: • wooly undercoat, from 1-2” in depth when the dog is in full coat • early warning system concerning burglaries or robberies • interest in jumping on furniture and sleeping in beds with humans • eye color of blue, brown, or one blue and one brown

24. Which option fits with stem? • Grammar issue – “a” only works with “wooly” • Also, author qualified answer in A – “when the dog is in full coat” – not all year round. The other options aren’t qualified

25. 6. Alaskan Malamutes’ tails are: • wiry • short • tightly curled • plumed and carried over their backs most of the time

26. Where did author spend time? • Correct answer is longest • Correct answer is most detailed • Correct answer is qualified

27. 7. Which of the following is char-acteristic of Alaskan Malamutes? • They don’t shed • They are always strong and silent • They can survive extreme cold and usually tolerate warm climates • They are always ‘one man’ dogs

28. Where did author spend time? • Correct answer is longest • Correct answer is qualified • Other answers include absolutes

29. Could these clues exist in your examinations?

30. Thrombotic microangiopathies are associated with: • Thrombocytopenia • Reduced coagulation factors • Antibodies to glomerula basement membranes • Anti-neutrophilic cytoplasmic antibodies • Necrotizing vasculitis

31. A patient carefully reads the labels on all her food products and calculates that she is taking in about 3.5 grams of sodium. This is closest to: • 60 mEq of sodium • 100 mEq of sodium • 160 mEq of sodium • 200 mEq of sodium

32. Numeric options • You may use C as the correct answer • It shouldn’t ALWAYS be the correct answer

33. 4. 3 formats for questions • Let’s talk about the 3 formats that you use for the test questions: • Multiple choice questions (MCQs) • True/False • Matching

34. Multiple-choice questions The examinee’s task is to choose the one best answer of 5 options (A-E)

35. Multiple Choice • Idea • Correct answer • Distractors

36. Multiple Choice Qs : Ideas • The idea for the question is sometimes called the ‘teaching point’ • You need questions for each lecture hour you present • Test essential concepts

37. Source of ideas • Essential content • Questions asked during lecture • Conversations overheard • Any points upon which you will build

38. How to start? • Review syllabus notes • What are the most essential concepts in the lecture? • The purpose of this lecture is to teach _______________________ • You might have 6 concepts, or you might have 2 concepts and some sub-topics

39. Sampling • Testing is always a sampling of knowledge • Even the USMLE samples knowledge • If you make a list of 12 things that MUST be tested from a lecture, perhaps ½ of them could be used now and ½ of them could be used later. (More on this in Topic 5 – Investing in the Future)

40. Characteristics of GREAT questions • Clearly written • Examinee knows what is being asked before reading the options • All the information needed to answer is in the question (options don’t build upon each other) • Simplest phrasing possible • No extraneous information

41. MCQs: Correct answer • Each question should have one answer that is best • When you’re writing, the best sequence is • Teaching point (What you want to ensure they know) • Correct answer • Distractors

42. Other Sequence • (Not recommended) • Vague concept • List of options • Choose one to be correct OR revise some options to be either correct or incorrect

43. Example: FK506 (Tacrolimus): • is an anti-proliferative drug • binds and inhibits the stimulation of a calcium activated phosphatase, thereby blocking the induction of the early response genes in the immune response • is less commonly used than cyclosporin because of its lower potency • is preferred over cyclosporin since it has no nephrotoxicity • is most commonly used to treat auto-immune disorders

44. If you didn’t know anything about FK506…. • Which option would you choose • Hint: retrace the steps of the question writer

45. FK506 (Tacrolimus): • is an anti-proliferative drug • binds and inhibits the stimulation of a calcium activated phosphatase, thereby blocking the induction of the early response genes in the immune response • is less commonly used than cyclosporin because of its lower potency • is preferred over cyclosporin since it has no nephrotoxicity • is most commonly used to treat auto-immune disorders

46. What to do? • Choose a specific teaching point from the lecture notes – what is it that you want to be sure the students know about FK506? • Form that teaching point into a question • Write the correct answer • Write the distractors

47. Multiple Choice Qs: Distractors The distractors should be: - plausible - less correct than the answer They are called ‘distractors’ because their intent is to DISTRACT examinees who do not know the concept

48. Sources for Distractors • Common misconceptions (“Why do some of the students think that ___________ “) • Logical thinking, but it doesn’t apply in this case

49. FK506 (Tacrolimus) binds and inhibits the stimulation of: • calcium activated phosphatase

50. Better format • But is it an essential teaching point?