"The Design and Use of Multiple Choice Tests"

"The Design and Use of Multiple Choice Tests"

Use of Multiple Choice Exams Not limited to Factual Knowledge • Making inferences • Application of models • Evaluation

Multiple Choice vs. Descriptive Examinations

Structure of a Multiple Choice Question Stem Answers

Formulating GoodMultiple Choice Questions • Wording the stem so that there is a single correct answer • Finding plausible wrong answers • Sensitivity to ESL students • Negative voice in the stem • Conditional “if… would” structures • Cultural contexts for questions

Stems Statements “A river that flows through a desert is called..” Questions “ Which of the following descriptions best fits the character of Alsatian wines?” Negative voice “ Which of the following grapes is not used to make Amarone?”

Stems Conditional Voice “ If you want a VQA Niagara red wine that was very fruity and required only a short ageing period you would select: ”

Answer Options • Single correct answer options • Collective answer options: “ (a) and (c) are both correct ” “ all of the above are correct ” “ none of the above are correct ”

Plausible Answer Strategies • Good questions contain several plausible answer options • Grey area between designing “tricky” questions and providing answer options that test knowledge thoroughly. • Inadvertently giving away the answers

Adding Uncertainty.. The term "fortified wine" means: a) the bottle had been reinforced b) the wine has had tannins and acid added c) the wine has had sugar added d) the wine has had SO2 added e) none of the foregoing

Adding difficulty.. Choose the best answer. Rootham’s, a well-known local manufacturer of brandied fruit preserves sells their product through gourmet food shops and at craft fairs throughout Canada. Their recipes use genuine brandy and all natural ingredients. They are able to sell these legally because: (a) the alcohol used is produced by fermentation (b) the product is made with ethyl alcohol (c) the alcohol used is purchased from the LCBO (d) the alcohol concentration is less than 1.2% (e) the label says “made with genuine brandy”

Making Multiple Choice Questions Interesting… A gracious, but parsimonious hostess wants to receive guests in the private dining room of your restaurant with a glass of champagne just to tantalise the taste buds and set the mood for the evening. 1.You suggest that if the wine is being poured at the bar, she should consider serving a Canadian sparkling wine that is very similar to Champagne in quality but considerably less expensive. Such a wine would be a:

2. The cheapest sparkling wine would be obvious to sophisticated guests because they would know that in the best sparkling wines… 3. If the gracious hostess insists on having “French” Champagne and you are left to rationalise costs, the most cost effective solution is to serve Champagne from..

4. Occasionally, a patron asks you to do something that will distinguish their event from others. One client wants to obtain a large bottle of Champagne to celebrate a particular event. Which is the largest bottle of champagne on the list below? 5. If someone wanted to serve a "glass of bubbly" during a reception, which of the following would be the most inappropriate type of wine to select as the pre-prandial (before dinner) libation?

Evaluating Multiple Choice Questions Difficulty Factor = R/N R = # of times the correct answer is selected N = # of people taking the exam Too Easy Too Difficult 0 0.5 + 1

Difficulty Factor Values greater than 0.8: • The question is too easy • Stem wording gives away the answer • Not enough plausible incorrect answer options Values less than 0.2: • Guessing effect can be a factor • The question is defective • Material wasn’t covered • Stem wording is unclear, confusing • Answer options need work

Evaluating Multiple Choice Questions Discrimination Index = Proportion of capable students that select an answer option – proportion of weak students that select the answer option i.e. D = Ftop quarter – Fbottom quarter Good Discrimination Reverse Discrimination - 1 0 + 1

Discrimination Index For the Correct Answer Option Values ~ 0.5 give good discrimination < 0.2 consider dropping negative… drop the question For Incorrect Answer Options Values should be negative.

Teaching WithMultiple Choice Questions • Prepare lectures incorporating all material covered in the questions to be used • After each lecture, check coverage by reviewing questions • Provide students with multiple choice questions for self-evaluation at the end of readings

Teaching WithMultiple Choice Examinations Use stem questions to teach additional or ancillary terms not covered in the course: i.e. …. For post-prandial (after dinner) drinks you want to stock a supply of fortified wines and brandy. The term "fortified wine" means….

Provide Students the Stem Questions Prior to the Exam! • Exam questions should cover all the course material being tested • Provides more detail than a course outline • Students focus their study effort on the topics they understand the least • Increases learning • Reduces stress

. Adapted from a presentation by J.E. (Joe) Barth School of Hospitality and Tourism Management University of Guelph, Ontario CANADA

"The Design and Use of Multiple Choice Tests"