question writing workshop stanford university medical education faculty l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Question Writing Workshop Stanford University Medical Education Faculty PowerPoint Presentation
Download Presentation
Question Writing Workshop Stanford University Medical Education Faculty

Loading in 2 Seconds...

play fullscreen
1 / 101

Question Writing Workshop Stanford University Medical Education Faculty - PowerPoint PPT Presentation


  • 430 Views
  • Uploaded on

Question Writing Workshop Stanford University Medical Education Faculty . Lynn C. Webb, Ed.D. August 15 , 2007. Agenda. Introductions Examination goals Giving away the answers 3 formats for questions Investing in future examinations The joy of feedback Questions / discussion.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Question Writing Workshop Stanford University Medical Education Faculty' - salena


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
question writing workshop stanford university medical education faculty

Question Writing WorkshopStanford University Medical Education Faculty

Lynn C. Webb, Ed.D.

August 15, 2007

testing@lwebb.com

agenda
Agenda
  • Introductions
  • Examination goals
  • Giving away the answers
  • 3 formats for questions
  • Investing in future examinations
  • The joy of feedback
  • Questions / discussion
1 introductions
1. Introductions
  • Lynn Webb – Testing consultant
  • Previous affiliations
    • American Board of Psychiatry & Neurology
    • National Board of Medical Examiners
    • ETS (Princeton)
    • ACT (Iowa City)
examples of tests
AAN’s RITE

ACP’s PRITE

APA’s FOCUS

AOA’s UHM

ABPM

AAPM&R’s SAE-R

WIP

CBNC

ABPN

Certification

Recertification

Subspecialty

Added qualifications

Non-medical exams

NCLEX

Solar Energy

Examples of Tests
2 examination goals
2. Examination Goals
  • Test students’ learning
  • Validate your teaching
  • Prepare students for future testing

Your questions can contribute to these goals, or detract from them.

contribute to goals
Contribute to Goals
  • Questions should be aligned with the specific content
  • Questions should be technically sound
    • Clearly written
    • Straight-forward, NOT tricky
detract from goals
Detract from Goals
  • Questions don’t address teaching or learning
  • Questions are convoluted or confusing, testing something other than the content
  • The focus shifts to the question instead of the content
3 giving away the answers
3. Giving away the answers
  • Medical students tend to be test-wise
  • When they don’t know an answer, they retrace your steps in writing the question to guess the correct answer
  • Even if YOU are test-wise, blind-spots exist when it’s your material
  • Multiple-choice questions are especially vulnerable
let s take a quiz
Let’s take a quiz
  • Our content will be highly esoteric facts about the Alaskan Malamute
  • Each quiz question will contain a flaw in its structure that will assist test-wise examinees. You won’t need to study the Alaskan Malamute to pass this quiz.
slide11
1. The primary function of sled dogs such as the Alaskan Malamute, Samoyed, Siberian Husky, and Eskimo Dog is to:
  • rescue drowning humans
  • retrieve tossed objects
  • guard property
  • pull sleds
test wise examinees
Test-wise examinees
  • Look for hints that the question writer overlooked between the question and the correct option
  • Sled dogs – Pull sleds (no other options about sleds)
slide13
2. Sled dogs provided vital transport of a life-saving serum during a diphtheria epidemic in Nome, Alaska in:
  • 1492
  • 1776
  • 1925
  • 1952
several ways to approach question
Several ways to approach question
  • First, eliminate options that aren’t plausible: 1492 and 1776 are too early for question content
  • Now you have 50/50 chance between C and D.
  • Is D plausible? Why not fly in 1952?
  • Numeric options – when in doubt, choose C
3 credit for the introduction of the alaskan malamute in the lower 48 is usually given to
3. Credit for the introduction of the Alaskan Malamute in the ‘lower 48’ is usually given to:
  • Arthur Treadwell Walden and The Seeleys
  • Admiral Byrd and Arthur Treadwell Walden
  • The Seeleys and Commodore Perry
  • Arthur Treadwell Walden and Jack London
look for repeats
Look for repeats
  • Arthur Treadwell Walden and the Seeleys
  • Admiral Byrd and Arthur Treadwell Walden
  • The Seeleys and Commodore Perry
  • Arthur Treadwell Walden and Jack London
more repeats
More repeats?
  • Arthur Treadwell Walden and The Seeleys
  • Admiral Byrd and Arthur Treadwell Walden
  • The Seeleys and Commodore Perry
  • Arthur Treadwell Walden and Jack London
some guys listed only once
Some guys listed only once
  • Arthur Treadwell Walden and The Seeleys
  • Admiral Byrd and Arthur Treadwell Walden
  • The Seeleys and Commodore Perry
  • Arthur Treadwell Walden and Jack London
where do repeats converge
Where do repeats converge?
  • Arthur Treadwell Walden and The Seeleys
  • Admiral Byrd and Arthur Treadwell Walden
  • The Seeleys and Commodore Perry
  • Arthur Treadwell Walden and Jack London
why do test wise examinees do that
Why do test-wise examinees do that?
  • They are retracing your steps in creating the question
  • That’s why you shouldn’t submit your first draft
  • It’s easy to fix this problem
4 champion coldfoot oonanik the world s most titled dog was how many inches at the withers
4. Champion Coldfoot Oonanik, the world’s most titled dog, was how many inches at the withers?
  • 2.75”
  • 25.0”
  • 27.5”
  • 275”
test wise examinees compare options
Test-wise examinees compare options
  • 3 options have the digits 2 – 7 – 5
  • One option has different digits (B), but it’s close in size to C
  • C must be the answer
  • (Author had the answer 27.5, then moved the decimal place for 2 distractors, then chose another plausible one near answer)
alaskan malamutes are known for a
Alaskan Malamutes are known for a:
  • wooly undercoat, from 1-2” in depth when the dog is in full coat
  • early warning system concerning burglaries or robberies
  • interest in jumping on furniture and sleeping in beds with humans
  • eye color of blue, brown, or one blue and one brown
which option fits with stem
Which option fits with stem?
  • Grammar issue – “a” only works with “wooly”
  • Also, author qualified answer in A – “when the dog is in full coat” – not all year round. The other options aren’t qualified
6 alaskan malamutes tails are
6. Alaskan Malamutes’ tails are:
  • wiry
  • short
  • tightly curled
  • plumed and carried over their backs most of the time
where did author spend time
Where did author spend time?
  • Correct answer is longest
  • Correct answer is most detailed
  • Correct answer is qualified
7 which of the following is char acteristic of alaskan malamutes
7. Which of the following is char-acteristic of Alaskan Malamutes?
  • They don’t shed
  • They are always strong and silent
  • They can survive extreme cold and usually tolerate warm climates
  • They are always ‘one man’ dogs
where did author spend time28
Where did author spend time?
  • Correct answer is longest
  • Correct answer is qualified
  • Other answers include absolutes
thrombotic microangiopathies are associated with
Thrombotic microangiopathies are associated with:
  • Thrombocytopenia
  • Reduced coagulation factors
  • Antibodies to glomerula basement membranes
  • Anti-neutrophilic cytoplasmic antibodies
  • Necrotizing vasculitis
slide31

A patient carefully reads the labels on all her food products and calculates that she is taking in about 3.5 grams of sodium. This is closest to:

  • 60 mEq of sodium
  • 100 mEq of sodium
  • 160 mEq of sodium
  • 200 mEq of sodium
numeric options
Numeric options
  • You may use C as the correct answer
  • It shouldn’t ALWAYS be the correct answer
4 3 formats for questions
4. 3 formats for questions
  • Let’s talk about the 3 formats that you use for the test questions:
    • Multiple choice questions (MCQs)
    • True/False
    • Matching
multiple choice questions
Multiple-choice questions

The examinee’s task is to choose the one best answer of 5 options (A-E)

multiple choice
Multiple Choice
  • Idea
  • Correct answer
  • Distractors
multiple choice qs ideas
Multiple Choice Qs : Ideas
  • The idea for the question is sometimes called the ‘teaching point’
  • You need questions for each lecture hour you present
  • Test essential concepts
source of ideas
Source of ideas
  • Essential content
  • Questions asked during lecture
  • Conversations overheard
  • Any points upon which you will build
how to start
How to start?
  • Review syllabus notes
  • What are the most essential concepts in the lecture?
  • The purpose of this lecture is to teach _______________________
  • You might have 6 concepts, or you might have 2 concepts and some sub-topics
sampling
Sampling
  • Testing is always a sampling of knowledge
  • Even the USMLE samples knowledge
  • If you make a list of 12 things that MUST be tested from a lecture, perhaps ½ of them could be used now and ½ of them could be used later. (More on this in Topic 5 – Investing in the Future)
characteristics of great questions
Characteristics of GREAT questions
  • Clearly written
    • Examinee knows what is being asked before reading the options
    • All the information needed to answer is in the question (options don’t build upon each other)
    • Simplest phrasing possible
    • No extraneous information
mcqs correct answer
MCQs: Correct answer
  • Each question should have one answer that is best
  • When you’re writing, the best sequence is
    • Teaching point (What you want to ensure they know)
    • Correct answer
    • Distractors
other sequence
Other Sequence
  • (Not recommended)
  • Vague concept
  • List of options
  • Choose one to be correct OR revise some options to be either correct or incorrect
example fk506 tacrolimus
Example: FK506 (Tacrolimus):
  • is an anti-proliferative drug
  • binds and inhibits the stimulation of a calcium activated phosphatase, thereby blocking the induction of the early response genes in the immune response
  • is less commonly used than cyclosporin because of its lower potency
  • is preferred over cyclosporin since it has no nephrotoxicity
  • is most commonly used to treat auto-immune disorders
if you didn t know anything about fk506
If you didn’t know anything about FK506….
  • Which option would you choose
  • Hint: retrace the steps of the question writer
fk506 tacrolimus
FK506 (Tacrolimus):
  • is an anti-proliferative drug
  • binds and inhibits the stimulation of a calcium activated phosphatase, thereby blocking the induction of the early response genes in the immune response
  • is less commonly used than cyclosporin because of its lower potency
  • is preferred over cyclosporin since it has no nephrotoxicity
  • is most commonly used to treat auto-immune disorders
what to do
What to do?
  • Choose a specific teaching point from the lecture notes – what is it that you want to be sure the students know about FK506?
  • Form that teaching point into a question
  • Write the correct answer
  • Write the distractors
multiple choice qs distractors
Multiple Choice Qs: Distractors

The distractors should be:

- plausible

- less correct than the answer

They are called ‘distractors’ because their intent is to DISTRACT examinees who do not know the concept

sources for distractors
Sources for Distractors
  • Common misconceptions (“Why do some of the students think that ___________ “)
  • Logical thinking, but it doesn’t apply in this case
better format
Better format
  • But is it an essential teaching point?
slide51
(or) What happens when FK506 (Tacrolimus) binds and inhibits the stimulation of calcium activated phosphatase?
  • The induction of the early response genes in the immune response is blocked
why change format
Why change format?
  • When questions are written clearly, it’s more likely that responses will reflect examinees’ knowledge on this topic, rather than their care in reading the question
  • Typical of the type of questions they will see on USMLE and specialty certification examinations
usmle others do not use
USMLE & others do NOT use:
  • Which of the following is TRUE?
  • Which of the following is FALSE?
  • Each of the following statements about X is true EXCEPT:
which one is true or false
Which one is true (or false)
  • Is not a multiple-choice question format
  • It’s a multiple true/false format
  • Options tend to be heterogeneous (mixing apples and oranges) because the teaching point isn’t focused
true false questions
True/False Questions
  • + Much easier/faster to write
  • - Less information about what the students know
  • They have a 50/50 chance of answering correctly, just by answering (MCQs were only 20% chance)
when to write a t f question
When to write a T/F question?
  • When the content is essential, but shallow
  • When the multiple choice question didn’t work
fk506 tacrolimus58
FK506 (Tacrolimus):
  • is an anti-proliferative drug
  • binds and inhibits the stimulation of a calcium activated phosphatase, thereby blocking the induction of the early response genes in the immune response
  • is less commonly used than cyclosporin because of its lower potency
  • is preferred over cyclosporin since it has no nephrotoxicity
  • is most commonly used to treat auto-immune disorders
true false qs ideas
True/False Qs: Ideas

(same as Multiple Choice Qs)

  • Essential content
  • Questions asked during lecture
  • Conversations overheard
  • Any points upon which you will build
true false qs correct answer
True/False Qs: Correct answer
  • Be clear
  • Don’t be tricky
  • If something is generally true (say in 999 of 1,000 cases) do you want them to say TRUE for the 999 cases or FALSE for the 1 case?
examples
Examples
  • Menarche is usually the first sign of puberty in females
  • Endometrial adenocarcinoma typically presents as high stage disease.
great material for t f
Great material for T/F

Can be found in unfocused multiple choice questions

which statement is true about bladder carcinoma
Which statement is true about Bladder carcinoma?
  • Bladder carcinoma is not associated with smoking
  • Recurrence and progression are very rare after local excision of low grade papillary carcinomas
  • “Field effect” refers to the concept that toxins in wheat grain pesticides may cause prostate carcinoma
  • Bladder carcinoma is usually adeno-carcinoma
  • Squamous cell carcinoma is associated with chronic irritative conditions such as chronic catheterization
matching
Matching
  • A series of 2 or more questions that use the same set of up to 10 possible answers (listed once as a heading).
  • The answers may be used once, more than once, or not at all
  • There is only ONE correct answer per question
matching ideas
Matching: Ideas
  • Essential content
  • Several concepts that are alike – typically testing recognition more than application of knowledge
  • Lists of diseases, microorganisms, medications, etc.
matching correct answers
Matching: Correct answers
  • This format requires more thought about correct answers
  • After preparing a matching set, review each description again (left side) comparing to each of the options on the right. You must be sure that only ONE is the best answer
matching distractors
Matching: Distractors
  • There are typically fewer distractors to generate for Matching than for MCQs, because the other correct answers are included in the distractor list
  • Additional distractors that are added should be common misconceptions about descriptions on left or plausible-sounding alternatives from within the category of things listed.
5 investing in future examinations
5. Investing in Future Examinations
  • You might decide to work ahead for future examinations when you’re working on your assigned questions.
  • Work ahead when you have an abundance of ideas
  • “Clone” your questions
cloning multiple choice qs
Cloning Multiple Choice Qs

Especially easy to do when starting from a question that had a negative context (NOT, EXCEPT, etc.)

which one of the following obstetric risks is not commonly associated with mullerian anomalies
Which one of the following obstetric risks is NOT commonly associated with mullerian anomalies?
  • Intrauterine growth restriction (IUGR)
  • Pre-eclampsia
  • Malpresentation
  • Preterm labor
4 options
4 options
  • 3 are associated with mullerian anomalies
  • 1 is NOT associated with mullerian anomalies
create a positive item
Create a positive item

Which of the following obstetric risks is commonly associated with mullerian anomalies?

  • Pre-eclampsia (now a distractor, instead of correct answer)
  • Intrauterine growth restriction (IUGR) (now the correct answer)
  • (add distractor here)

D. (add distractor here)

clone new item
Clone new item
  • Same item, use same distractors, but insert new correct answer
which one of the following obstetric risks is commonly associated with mullerian anomalies
Which one of the following obstetric risks is commonly associated with mullerian anomalies?
  • Pre-eclampsia (still a distractor, instead of correct answer)
  • Malpresentation (new correct answer)
  • (add same distractor here)

D. (add same distractor here)

clone second item
Clone second item
  • Same process
  • Keep same distractors
  • Insert new correct answer
which one of the following obstetric risks is commonly associated with mullerian anomalies78
Which one of the following obstetric risks is commonly associated with mullerian anomalies?
  • Pre-eclampsia (still a distractor, instead of correct answer)
  • Preterm labor(new correct answer)
  • (add same distractor here)

D. (add same distractor here)

another example
Another example
  • Change negative context Multiple choice question to positive, then clone it.
negative context question
Negative context question

Post-streptococcal glomerulonephritis is associated with all EXCEPT

  • Subepithelial humps
  • Pharyngitis
  • Low serum complement levels
  • Low anti-streptolysin antibody titers*
  • Hematuria
post streptococcal glomerulonephritis is associated with
Post-streptococcal glomerulonephritis is associated with :
  • Low anti-streptolysin antibody titers (now a distractor)
  • Subepithelial humps (now correct)
  • (add a distractor)
  • (add a distractor)
  • (add a distractor)
clone 3 more items
Clone 3 more items
  • Keep the 4 distractors (1 old, 3 new) the same
  • Change the correct answer each time, alternatively using:
    • Pharyngitis
    • Low serum complement levels
    • Hematuria
do students share exams
Do students share exams?
  • Cloning items is a great way to discourage memorizing test materials. The correct answer changes across administrations of the test.
many topics have more than one compenent
Many topics have more than one compenent
  • Think of the questions you used to write as “All of the above” - Those kinds of topics are great for cloning
  • Write the question with only one of the possible correct answers
  • Work hard to develop plausible distractors
  • Keep the question and distractors; alternate the correct answers
use 1 per test
Use 1 per test
  • Store the others for the future
cloning true false
Cloning True/False
  • Write the True/False question
  • Clone it to the opposite answer
true original
TRUE Original
  • The most common cause of congenital adrenal hyperplasia is 21-hydroxylase deficiency
  • Now, clone to FALSE
clone to false
Clone to FALSE

The most common cause of congenital adrenal hyperplasia is 3-beta-HSD deficiency

Or

The most common cause of congenital adrenal hyperplasia is 11-hydroxylase deficiency

cloning matching
Cloning Matching
  • Not as easy to do as for Multiple Choice or True/False
  • Easiest when the list of possibilities is very long
  • Try to utilize the options that were not used in the original set, if possible.
6 the joy of feedback
6. The joy of feedback
  • Writing questions is difficult
  • Writing questions alone is even more difficult
  • It’s unlikely that your first draft of any question should be submitted
clinical presentation was here
(Clinical presentation was here)
  • Which one of the following is the best explanation for the serum sodium concentration and the concentrated urine?
options in first draft
Options in first draft
  • Arginine vasopressin release caused by hypotonic serum.
  • Arginine vasopressin release caused by hypertonic urine.
  • Arginine vasopressin release caused by low effective blood volume.
  • Arginine vasopressin release caused by a combination of hypertonic urine and low effective blood volume
your colleague recommends tidying the options
Your colleague recommends tidying the options
  • The phrase “Arginine vasopressin release caused by” repeats in all the options
  • Move it up to the question
slide94
The best explanation for the serum sodium concentration and the concentrated urine is that arginine vasopressin release was caused by:
  • hypotonic serum
  • hypertonic urine
  • low effective blood volume
  • a combination of hypertonic urine and low effective blood volume
1 st drafts can include too many details
1st drafts can include too many details
  • When it’s your patient, many of the details will come to mind when writing the test question
  • It’s easier for your colleague to weed out any information that isn’t needed to answer the question
sample first draft question
Sample first draft question

Miss Smith, a previously healthy 22-year-old woman, is brought to the ER by her overly-protective mother. Miss Smith has a one day history of dysuria, urinary frequency, urgency, and supra-pubic tenderness. She noted cloudy urine, but denied fever. Her urinalysis showed many polymorphonuclear leukocytes and bacteria. Which of the following best characterizes her condition?

edited question
Edited question

Miss Smith, a previously healthy 22-year-old woman, presents to the ER with a one day history of dysuria, urinary frequency, urgency, and supra-pubic tenderness. She noted cloudy urine, but denied fever. Her urinalysis showed many polymorphonuclear leukocytes and bacteria. Which of the following best characterizes her condition?

thank you

THANK YOU

Lynn C. Webb, Ed.D.

testing@lwebb.com