Writing exam questions in the clinical sciences
Advertisement
This presentation is the property of its rightful owner.
1 / 38

Writing Exam Questions in the Clinical Sciences PowerPoint PPT Presentation

Writing Exam Questions in the Clinical Sciences. Faculty Professional Development Series University of Pennsylvania School of Medicine November 17, 2004 Jennifer R. Kogan, M.D. Judy A. Shea, Ph.D. Department of Medicine. Materials adapted from the National Board of Medical Examiners. Agenda.

Download Presentation

Writing Exam Questions in the Clinical Sciences

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Writing exam questions in the clinical sciences

Writing Exam Questions in the Clinical Sciences

Faculty Professional Development SeriesUniversity of Pennsylvania School of MedicineNovember 17, 2004

Jennifer R. Kogan, M.D.

Judy A. Shea, Ph.D.

Department of Medicine

Materials adapted from the National Board of Medical Examiners


Agenda

Agenda

Review item types

Review structure of A type questions

Review technical item flaws

Analyze submitted items

Wrap-up


Steps in test development

Steps in Test Development

Test purpose

Testing time and method of administration

Test standardization

Test content

Item format

Number of items

Developing items

Item selection and evaluation


Overview of item types

Overview of Item Types

True-false

  • C (A/B/Both/Neither)

  • K (complex true/false)

  • X (simple true/false)

  • simulations such as PMPs

    One-best answer

  • A (4 or more options)

  • B (4 or 5 option matching sets in sets of 2–5 items)

  • R (extended matching items in sets of 2-20 items)


True false question x type

True/false question X-type

Which of the following is/are X-linked recessive conditions?

1. Hemophilia A (classic hemophilia)T

2. Cystic fibrosis F

3. Duchenne’s muscular dystrophyT

4. Tay-Sachs disease F


True false questions

Advantages

simple direct test of knowledge

efficient

easy to write

Disadvantages

each statement must be clearly true/false

test trivia

encourage memorization

ambiguous

susceptible to guessing

True-false Questions

Avoid true-false questions


Single best answer a type

Single Best Answer A-type

A previously healthy 15-year-old boy has cramping periumbilical pain; after several hours, the pain shifts to the right lower quadrant and becomes constant. He vomits several times and is brought to the emergency department. The abdomen is tender on deep palpation of the right lower quadrant. Findings on chest and abdominal x-ray films are normal. Leukocyte count is 15,000/mm3. Urinalysis shows 3 leukocytes/hpf. Which of the following is the most appropriate initial management?

A. Supportive treatment at home; return at once if the pain increases

B. Barium enema

C. CT scan of the abdomen

D. Intravenous pyelography and cystography

E. Surgical exploration of the abdomen*


Extended matching question r type

Extended Matching Question R-type

A. Left anterior cerebral artery E. Left posterior cerebral artery

B. Right anterior cerebral artery F. Right posterior cerebral artery

C. Left middle cerebral artery G. Left lenticulostriate arteries

D. Right middle cerebral artery H. Right lenticulostriate arteries

For each patient with neurological abnormalities, select the artery that is

most likely to be involved.

1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response on the right, normal strength of the right arm, and normal facial movements. Answer: A

2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and paralysis of the lower two-thirds of his face on the right. His speech is fluent, and he has normal comprehension of verbal and written commands. Answer: G


Item type answer implications

Item Type Answer Implications

True-false

  • absolute/non-debatable

    One-best answer

  • there is one better/best answer

True

False

C

A

D

E

B


Components of the a type question

Components of the A-type Question

Stem:A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg.

Lead-in:Which of the following muscles is most likely to have been injured?

Options:A. Gluteus maximus*

B. Gluteus minimus

C. Hamstrings

D. Iliopsoas

E. Obturator internus

Distractors


Rules for writing a type questions

Rules for Writing A-type Questions

1) Focus on an important topic, usually a common or

critical clinical problem; avoid esoterica and “zebras”

2) Assess application of knowledge, not recall

3) Pose clinical decision-making tasks that are within the

education/experience of examinees

4) Pose a clear question in the lead-in

  • can you answer it without looking at the options?

    5) Use homogeneous distractors

    6) Avoid technical flaws


Tools

Patient vignettes should

include

age, gender

site of care

presenting complaint

duration

patient history

physical findings

+/- diagnostic studies

+/- initial treatment

Stems should

not be completely based on real patients

include reference material when it would be realistic in practice

not use the patient’s or doctor’s own words

not include patients who lie

Tools


Lead ins

Lead-Ins

Health maintenance

  • Which of the following is the most appropriate screening test?

  • Which of the following immunizations should be administered at this time?

    Mechanisms of disease

  • Which of the following is the most likely pathogen?

  • Which of the following is the most likely explanation for the findings?


Exam questions

Diagnosis

  • Which of the following is the most likely diagnosis?

  • Which of the following is the most appropriate next step in diagnosis?

    Management

  • Which of the following is the most appropriate next step in patient care?

  • Which of the following is the most effective management?


Technical item flaws

Technical Item Flaws

Issues related to

Irrelevant Difficulty

Issues Related to

Testwiseness


Grammatical cues

Grammatical Cues

The option(s) does not flow from the stem

The minor differences among organisms of the

same kind are known as

A. Heredity

B. Variations

C. Adaptation

D. Natural selection


Logical cues

Logical Cues

A subset of options are collectively exhaustive.

Crime is

A. Equally distributed among the social classes

B. Overrepresented among the poor

C. Overrepresented among the middle class and rich

D. Primarily an indication of psychosexual maladjustment

E. Reaching a plateau of tolerability for the nation


Absolute terms

Absolute Terms

Terms such as ‘always’ or ‘never’ are used in options.

In patients with advanced dementia, Alzheimer’s type, the memory defect

A. Can be treated adequately with lecithin

B. Could be a sequela of early parkinsonism

C. Is never seen in patients with neurofibrillary tangles

C. Is never severe

D. Possibly involves the cholinergic system


Long correct answer

Long Correct Answer

The correct answer is longer, more specific, or more

complete than the other options.

Secondary gain is

A. Synonymous with malingering

B. A frequent problem in obsessive-compulsive disorder

C. A complication of a variety of illnesses and tends to prolong many of them

D. Never seen in organic brain damage


Word repeats

Word Repeats

A word or phrase is included in the stem and correct

answer.

A58-year-old man with a history of heavy alcohol

use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called

A. Depersonalization

B. Derailment

C. Derealization*

D. Focal memory defect


Convergence

Convergence

The correct answer includes the most elements in

common with the other options

Local anesthetics are most effective in the

A. Anionic form, acting from inside the nerve membrane

B. Cationic form, acting from inside the nerve membrane*

C. Cationic form, acting from outside the nerve membrane

D. Uncharged form, acting from inside the nerve membrane

E. Uncharged form, acting from outside the nerve membrane


Options are long complicated or doubled

Options are long, complicated or doubled

Systematic geography differs from regional geography in that

A. Systematic geography deals, in the main, with physical geography, while regional geography concerns itself essentially with the field of human geography

B. Systematic geography studies a region systematically while regional geography is concerned only with descriptive account of a region

C. Systematic geography studies a single phenomenon in its distribution over the earth in order to supply generalizations for regional geography, which studies the arrangement of phenomena in one given area*


Numeric data are not stated consistently

Numeric data are not stated consistently

Following a second episode of infection, what is the the

likelihood that a woman is infertile?

A. Less than 20%

B. 20% to 30%

C. Greater than 50%

D. 90%

E. 75%


Frequency terms in the options are vague

Frequency terms in the options are vague

Severe obesity in early adolescence

  • Usually responds dramatically to dietary regimens

  • Often is related to endocrine disorders

  • Has a 75% change of clearing spontaneously

  • Shows a poor prognosis

  • Usually responds to pharmacotherapy and intensive psychotherapy


Language in the options is not parallel

Language in the options is not parallel

In a vaccine trial, 200 2-year-old boys were given a vaccine

against a certain disease and then monitored for five years for

occurrence of disease. Of this group, 85% never contracted the disease.

Which of the following statements concerning these results is correct?

  • No conclusions can be drawn since no follow-up was made of non-vaccinated children

  • The number of cases (I.e. 30 cases over five years) is too small for statistically meaningful conclusions

  • No conclusions can be drawn because the trial involved only boys

  • Vaccine efficacy (%) is calculated as 85-15/100


Fixed

Fixed

In a vaccine trial, 200 2-year-old boys were given a vaccine

against a certain disease and then monitored for five years for

occurrence of disease. Of this group, 85% never contracted the disease.

For which of the following reasons can no conclusion be drawn from

these results?

  • No follow-up was made of non-vaccinated children

  • The number of cases (I.e. 30 cases over five years) was too small

  • The trial involved only boys

  • Write new option


Options in an nonlogical order

Options in an nonlogical order

The population of Denmark is

A. 2 million

B. 15 million

C. 4 million

D. 7 million


None of the above is used as an option

“None of the Above” is used as an option

Which city is closest to New York City?

A. Boston

B. Chicago

C. Dallas

D. Los Angeles

E. None of the above


Window dressing and red herrings

“Window Dressing” and “Red Herrings”


Non vignette

The most likely renal abnormality

in children with nephrotic

syndrome and normal renal

function is

Acute poststreptococcal glomerulonephritis

Hemolytic-uremic syndrome

Minimal change disease

Focal and segmental glomerulosclerosis

Schonlein-Henoch purpura

Non-vignette


Short vignette

A 2 year old boy has a 1

week history of edema. His

blood pressure is 100/60

mmHg and there is

generalized edema and

ascites. Labs show Cr 0.4

mg/dL, albumin 1.4 g/dL and

cholesterol of 569 mg/dL.

UA shows 4+ protein and no

blood. The most likely

diagnosis is

Short vignette


Long vignette

A 2 year old black child developed

swelling of his eyes and ankles

over the past week. Blood pressure

is 100/60 mmHg, pulse 110/min

respirations 28/min. Exam shows

swelling of eyes, abdominal

distention and a positive fluid

Labs show Cr 0.4

mg/dL, albumin 1.4 g/dL and

cholesterol of 569 mg/dL.

UA shows 4+ protein and no

blood. The most likely

diagnosis is

Long Vignette


Analysis of submitted items

Analysis of Submitted Items


Steps in writing r types

Start with the theme

Fatigue

List all diagnoses

Acute leukemia

Congestive heart failure

Depression

Epstein-Barr virus

Folate deficiency

Glucose 6-phosphate deficiency

And so on…

Write vignettes

A 15-year-old girl has a two week history of fatigue and back pain. She has widespread bruising, pallor, and tenderness over the vertebrae and both femurs. Complete blood count shows hemoglobin concentration of 7.0 g/dL, leukocyte count of 2000/mm, and platelet count of 15,000/mm

Steps in Writing R-types


Some tips for writing r types

Some Tips for Writing R-types

More than one vignette can be prepared for common

treatable conditions

Shorter/longer vignettes can be used

It is easy to convert them to one-best-answer questions and vice versa

There needs to be a single best answer for each stem

All rules/technical flaws for single-best answer questions apply


Sample lead ins and topics for r types

Sample Lead-ins and Topics for R-types

For each of the following patients, select the most likely (cause).

  • Underlying mechanism of disease, medications, toxic agents…

    For each of the following patients with (chief complaint), select the most likely diagnosis.

  • Lists of diagnoses

    For each of the following patients, select the (finding) that would be expected.

  • Laboratory results, physical signs…


Take home points

Take Home Points

  • Each item should focus on an important concept

  • Each item should assess application of knowledge, not recall of an isolated fast

  • The stem of the item must pose a clear question, and it should be possible to arrive at the answer with the options covered

  • All distractors should be homogeneous

  • Avoid technical item flaws that provide specific benefit to testwise examinees or that pose irrelevant difficulty.


Suggested reading

Suggested Reading

  • Basic texts

    • Case, .S and Swanson, D.B. Constructing Written Test Questions for the Basic and Clinical Sciences, NBME, 2003.

      http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf

    • Crocker, L and Algina, J. Introduction to Classical and Modern Test Theory. Holt, Rinehart and Winston, 1986.

    • Ebel, R.L. Measuring Educational Achievement. Prentice Hal, 1965.

    • Haladyna, T.M. Developing and Validating Multiple-Choice Test Items. Laurence Erlbaum Associates, Inc.

  • Advanced texts

    • Baker, F.B. The Basics of Item Response Theory. Heinemann, 1985.

    • Linn, R.L. (Ed.) Educational Measurement. Macmillan Publishing Company, 1989.

    • Wainer, H. and Braun, H.I. (Eds.) Test Validity. Lawrence Erlbaum, 1988.


  • Login