david dibattista ph d brock university psychology department n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
David DiBattista, Ph.D. Brock University Psychology Department PowerPoint Presentation
Download Presentation
David DiBattista, Ph.D. Brock University Psychology Department

Loading in 2 Seconds...

play fullscreen
1 / 108

David DiBattista, Ph.D. Brock University Psychology Department - PowerPoint PPT Presentation


  • 120 Views
  • Uploaded on

Getting the Most Out of Multiple-choice Questions. David DiBattista, Ph.D. Brock University Psychology Department. Overview. About multiple-choice testing Two challenges in MC testing Addressing the challenges. About MC Testing. Most common selected-response format

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'David DiBattista, Ph.D. Brock University Psychology Department' - vanna-petty


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
overview
Overview
  • About multiple-choice testing
  • Two challenges in MC testing
  • Addressing the challenges
about mc testing
About MC Testing
  • Most common selected-response format
  • Permits broad coverage of material
  • Practical even with large classes
about mc testing1
About MC Testing
  • Most common selected-response format
  • Permits broad coverage of material
  • Practical even with large classes
some important terminology
Some Important Terminology
  • The stem is a question or incomplete statement that is followed by several alternatives.
  • One alternative is the correct answer (or key) and the others are distracters.
  • The correct-answer format is typical, with the best-answerformat less common.
overview1
Overview
  • About multiple-choice testing
  • Two challenges in MC testing
  • Addressing the challenges
two challenges in mc testing
Two Challenges in MC Testing

Challenge #1

Constructing high-quality MC items takes time, effort, and skill.

Challenge #2

MC items are too often used to assess what students remember, with higher-level cognitive skills getting little attention.

overview2
Overview
  • About multiple-choice testing
  • Two challenges in MC testing
  • Addressing the challenges

1. Constructing high-quality items

2. Assessing higher-level thinking

tips for mc item construction
Tips for MC Item Construction
  • Use question format whenever possible and sentence-completion format only when necessary. In either case, use appropriate punctuation and capitalization.
slide11

When hungry rats are given small bits of food at fixed intervals, excessive water consumption occurs. What is this excessive consumption called?

A. primary drinking

B. osmometric drinking

C. psychogenic polydipsia

D. schedule-induced polydipsia

slide12

When hungry rats are given small bits of food at fixed intervals, excessive water consumption occurs. This excessive consumption is known as

A. primary drinking.

B. osmometric drinking.

C. psychogenic polydipsia.

D. schedule-induced polydipsia.

tips for mc item construction1
Tips for MC Item Construction
  • If you use completion format, always place the blank at the very end of the stem.
slide14

Poor

Generally speaking, ___________ testing supports content sampling better than do other testing formats.

A. essay

B. performance

C. multiple-choice

D. paper-and-pencil

slide15

Improved

Generally speaking, content sampling is best supported by

A. essay testing.

B. performance testing.

C. multiple-choice testing.

D. paper-and-pencil testing.

slide16

Better still

Generally speaking, which type of testing best supports content sampling?

A. essay testing

B. performance testing

C. multiple-choice testing

D. paper-and-pencil testing

tips for mc item construction2
Tips for MC Item Construction
  • The stem should present the issue under consideration clearly and contain as much information as possible.
  • Do not include irrelevant information in the stem unless it plays a role in the assessment procedure.
slide18

Poor

South America

A. imports coffee from Australia.

B. is where the Gobi Desert is found.

C. was colonized mostly by people from

Spain.

D. has a larger population than the United

States of America.

slide19

Even worse

South America,which has an area of more than 17 million square kilometres,

A. imports coffee from Australia.

B. is where the Gobi Desert is found.

C. was colonized mostly by people from

Spain.

D. has a larger population than the United

States of America.

slide20

Better

Most of South America was colonized by people from what country?

A. Spain

B. France

C. Holland

D. England

slide21

A distribution of 100 test scores is positively skewed, with a mean of 70 and a standard deviation of 10. John’s test score is 80. What is his standard score?

A. -1.0

B. +1.0

C. -10.0

D. +10.0

slide22

A distribution of 100 test scores is positively skewed, with a mean of 70 and a standard deviation of 10. John’s test score is 80. What is his standard score?

A. -1.0

B. +1.0

C. -10.0

D. +10.0

tips for mc item construction3
Tips for MC Item Construction
  • Check carefully for spelling errors, giving special attention to distracters.
  • If you use sentence-completion format, check carefully for grammatical consistency of stem and alternatives.
slide24

Poor

Stamp collectors use stamp hinges to A. pick up stamps.

B. fold learge stamps in half.

C. mount stamps in albums.

D. joining two stamps together.

slide25

Improved

Stamp collectors use stamp hinges to A. pick up stamps.

B. fold large stamps in half.

C. mount stamps in albums.

D. join two stamps together.

slide26

Better still

What do stamp collectors use stamp hinges for?

A. picking up stamps

B. folding large stamps in half

C. mounting stamps in albums

D. joining two stamps together

tips for mc item construction4
Tips for MC Item Construction
  • Whenever possible, avoid negative wording in the stem, and be sure to emphasize it when it does occur.
slide28

Poor

Which term is not usually associated with Sigmund Freud?

A. id

B. extinction

C. repression

D. latent content

slide29

Better

Which term is NOT usually associated with Sigmund Freud?

A. id

B. extinction

C. repression

D. latent content

slide30

Also better

Which term is usually associated with behaviourism?

A. id

B. extinction

C. repression

D. latent content

tips for mc item construction5
Tips for MC Item Construction
  • All distracters should be plausible.
  • Four alternatives will usually be quite adequate, but the number used is best determined by the number of plausible distracters you can supply.
slide32

Which river flows through the city of Edmonton?

A.North Saskatchewan River

B. Athabasca River

C. Peace River

D. Milk River

slide33

Which river flows through the city of Edmonton?

A. North Saskatchewan River

B. Mississippi River

C. Amazon River

D. Rhine River

E. Seine River

F. Nile River

tips for mc item construction6
Tips for MC Item Construction
  • To generate plausible distracters
  • Use students’ most common errors on constructed-response tests.
  • Use distracters that are similar to the correct answer in content, length, and complexity.
  • Use words that sound important or have associations to the stem.
  • Use distracters that are true, but do not correctly answer the question.
slide35

Name the river that flows through the city of Edmonton.

Athabasca River

What do stamp collectors use stamp hinges for?

To join stamps together

tips for mc item construction7
Tips for MC Item Construction
  • To generate plausible distracters
  • Use students’ most common errors on constructed-response tests.
  • Use distracters that are similar to the correct answer in content, length, and complexity.
  • Use words that sound important or have associations to the stem.
  • Use distracters that are true, but do not correctly answer the question.
slide37

In severe cases of obesity, there may be a substantial increase in the number of adipocytes. What term is used to refer to this increase?

A. hyperbole

B. hyperplasia

C. hypertrophy

D. hypertonicity

tips for mc item construction8
Tips for MC Item Construction
  • To generate plausible distracters
  • Use students’ most common errors on constructed-response tests.
  • Use distracters that are similar to the correct answer in content, length, and complexity.
  • Use words that sound important or have associations to the stem.
  • Use distracters that are true, but do not correctly answer the question.
slide39

In responding to a lengthy survey, a man answers “yes” to every yes-no question asked. It is reasonable to suspect that his responses may be influenced by which of the following?

A. response acquiescence

B. opportunistic characterization

C. the partial reinforcement effect

D. the conspicuous agreement predisposition

tips for mc item construction9
Tips for MC Item Construction
  • To generate plausible distracters
  • Use students’ most common errors on constructed-response tests.
  • Use distracters that are similar to the correct answer in content, length, and complexity.
  • Use words that sound important or have associations to the stem.
  • Use distracters that are true, but do not correctly answer the question.
slide41

Which of the following events caused the Prime Minister of Canada to proclaim the War Measures Act?

  • Newfoundland was occupied by German
  • forces in 1940.
  • B. The October Crisis occurred in 1970.
  • C. The Quebec Referendum was held in
  • 1980.
  • D. The Meech Lake Accord was defeated in
  • 1990.
tips for mc item construction10
Tips for MC Item Construction
  • Avoid patterns in the length and location of correct answers that could provide clues that are unrelated to content.
  • Balance the answer key so that the correct response appears in each position about the same number of times.
slide43

Your baseball team has a runner on first base. Under what circumstances might it be a good idea for the batter to lay down a sacrifice bunt?

  • when it is a night game
  • B. when the pitcher is a relief pitcher
  • C. when the batter is a right-handed hitter
  • D. when it is late in the game, the score is
  • tied, there are no outs, and the batter is a
  • weak hitter with less than two strikes
slide44

In a four-alternative multiple-choice test, about how often should the correct answer appear in each of the four locations?

A. 10% of the time

B. 25% of the time

C. 40% of the time

D. 60% of the time

slide45

Who invented the binaural recording system commonly known as “stereo”?

A. Xxxxxxxxxxxxxx

B. Xxxxxxxxxxxxxx

C. Xxxxxxxxxxxxxx

D. Xxxxxxxxxxxxxx

slide46

Who invented the binaural recording system commonly known as “stereo”?

A. Alan Dower Blumlein

B. Alan Dower Blumlein

C. Alan Dower Blumlein

D. Alan Dower Blumlein

tips for mc item construction11
Tips for MC Item Construction
  • Do not use “none of the above.”
slide48

“None of the above” as a distracter

Poor

Which of these 19th century authors wrote Wuthering Heights?

A. Emily Bronte

B. George Eliot

C. Charles Dickens

D. None of the above

slide49

“None of the above” as a distracter

Better

Which of these 19th century authors wrote Wuthering Heights?

A. Emily Bronte

B. George Eliot

C. Charles Dickens

D. William Thackeray

slide50

“None of the above” as the key

Poor

Which of these 19th century authors wrote Middlemarch?

A. Jane Austen

B. Anne Bronte

C. Willkie Collins

D. None of the above

slide51

“None of the above” as the key

Which of these 19th century authors wrote Middlemarch?

A. Jane Austen

B. Anne Bronte

C. Willkie Collins

D. Charles Dickens

Hey! Wait a minute!

Dickens didn’t write Middlemarch!

slide52

“None of the above” as the key

Better

Which of these 19th century authors wrote Middlemarch?

A. Jane Austen

B. Anne Bronte

C. Willkie Collins

D. George Eliot

tips for mc item construction12
Tips for MC Item Construction
  • Do not use “all of the above” unless there are only two distracters.
slide54

Poor

Which of these terms is associated with Sigmund Freud?

A. superego

B. repression

C. latent content

D.all of the above

slide55

Poor

Which of these terms is associated with Sigmund Freud?

A. superego 

B. repression

C. latent content

D. all of the above

slide56

Poor

Which of these terms is associated with Sigmund Freud?

A. superego 

B. repression 

C. latent content

D. all of the above

slide57

Poor

Which of these terms is associated with Sigmund Freud?

A. superego 

B. repression 

C. latent content???????

D. all of the above

slide58

Better

Which of these terms is associated with Sigmund Freud?

A. superego

B. bulimia nervosa

C. cognitive dissonance

D.fixed-interval schedule

slide59

Also better

Which of these terms is associated with Sigmund Freud?

A. superego

B. latent content

C. both of the above

tips for mc item construction13
Tips for MC Item Construction
  • Do not use structurally complex items (e.g., “Type K”), which may call heavily on test-taking skills.
slide61

Poor

  • Which of the following may threaten the internal validity of an experiment?
  • 1. Instrumentation changes
  • 2. Homogeneity of variance
  • 3. Differential participant mortality
  • 4. Non-random participant selection
  • A. 1 and 3 only
  • B. 2 and 4 only
  • C. 1, 2, and 3
  • D. 4 only
  • 1, 2, 3, and 4
slide62

Poor

  • Which of the following may threaten the internal validity of an experiment?
  • 1. Instrumentation changesThreat
  • 2. Homogeneity of varianceNo threat
  • 3. Differential participant mortality ???
  • 4. Non-random participant selection ???
  • A. 1 and 3 only
  • B. 2 and 4 only
  • C. 1, 2, and 3
  • D. 4 only
  • 1, 2, 3, and 4
slide63

Poor

  • Which of the following may threaten the internal validity of an experiment?
  • 1. Instrumentation changesThreat
  • 2. Homogeneity of varianceNo threat
  • 3. Differential participant mortality ???
  • 4. Non-random participant selection ???
  • A. 1 and 3 only
  • B. 2 and 4 only X (from knowing #2)
  • C. 1, 2, and 3 X (from knowing #2)
  • D. 4 only
  • 1, 2, 3, and 4 X (from knowing #2)
slide64

Poor

  • Which of the following may threaten the internal validity of an experiment?
  • 1. Instrumentation changesThreat
  • 2. Homogeneity of varianceNo threat
  • 3. Differential participant mortality ???
  • 4. Non-random participant selection ???
  • A. 1 and 3 only
  • B. 2 and 4 only X (from knowing #2)
  • C. 1, 2, and 3 X (from knowing #2)
  • D. 4 only X (from knowing #1)
  • 1, 2, 3, and 4 X (from knowing #2)
tips for mc item construction14
Tips for MC Item Construction
  • Avoid trivia. Test only important information, concepts, and abilities.
slide66

Peace talks were held in Paris after World War I. How long did these talks last?

A. 3 months

B. 6 months

C. 9 months

D. 12 months

slide67

Which of the following countries DID NOT join the League of Nations in the period following World War I?

A. United States

B. Germany

C. France

D.Japan

tips for mc item construction15
Tips for MC Item Construction

The last suggestion

  • Ignore any of the preceding suggestions when you have a good reason to do so.
overview3
Overview
  • About multiple-choice testing
  • Two challenges in MC testing
  • Addressing the challenges

1. Constructing high-quality items

2. Assessing higher-level thinking

two challenges in mc testing1
Two Challenges in MC Testing

Challenge #1

Constructing high-quality MC items takes time, effort, and skill.

Challenge #2

MC items are too often used to assess what students remember, with higher-level cognitive skills getting little attention.

slide72

The Original Bloom’s Taxonomy

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

assessing higher level thinking
Assessing Higher-level Thinking

Although MC tests too often focus on what students can remember, this can usually be avoided by

  • using challenging individual MC items

Ramp it up! (Bloom’s Revised Taxonomy)

Item shells

  • using interpretive exercises
slide77

According to the Empirical Rule, about how many scores will typically lie within one standard deviation on either side of the mean of a mound-like distribution?

A. one-tenth

B. one-quarter

C. one-half

D. two-thirds

slide78

The mean of a mound-like distribution of scores is 65 and the standard deviation is 12. You can be pretty sure that about two-thirds of the scores lie between what two values?

A. 59 and 71

B. 53 and 77

C. 47 and 83

D. 41 and 89

slide79

The mean of a mound-like distribution of scores is 65 and the standard deviation is 12. You can be pretty sure that about two-thirds of the scores lie between what two values?

A. 59 and 71

B. 53 and 77

C. 47 and 83

D. 41 and 89

slide80

The mean of a mound-like distribution of scores is 65 and the variance is 144. You can be pretty sure that about two-thirds of the scores lie between what two values?

A. 59 and 71

B. 53 and 77

C. 47 and 83

D. 41 and 89

slide81

What characteristic of a distribution does the standard deviation measure?

A. dispersion

B. skewness

C. normality

D. kurtosis

slide82

Median is to variance as central tendency is to

A. kurtosis.

B. normality.

C. skewness.

D. dispersion.

slide83

In classical conditioning, what name is give to a stimulus that elicits a particular response even in the absence of any prior training?

A. conditioned stimulus

B. unconditioned stimulus

C. activational stimulus

D. discriminative stimulus

slide84

Right after a rat smells menthol, it is always given Drug X, which causes osmometric drinking. Eventually, the rat drinks whenever it smells menthol, even when it is not injected with Drug X. In this situation, what is the role of Drug X?

A. conditioned stimulus

B. unconditioned stimulus

C. activational stimulus

D. discriminative stimulus

slide85

Which name is most closely associated with classical psychoanalytic therapy?

  • B.F. Skinner
  • Abraham Maslow
  • Leon Festinger
  • D. Sigmund Freud
slide86

What must be done to the following sentence to make it accurate?

  • Classical psychoanalytic therapy, which was developed by Freud and involves extensive use of free association, is more widely used today than it was 60 years ago.
  • Nothing needs to be done.
  • Change the word “Freud” to “Skinner.”
  • Change the words “free association” to
  • “behaviour modification techniques.”
  • D. Change the word “more” to “less.”
slide87

Claim: Captain Queeq was a coward. Which of the following most strongly supports this claim?

A. his bullying of Stillwell

B. his behaviour at Saipan

C. the cutting of the towline

D. the affair of the strawberries

slide88

Your friend Anne reads that census data indicate that people are having fewer children nowadays than they did 40 years ago. She tells you that she does not believe this because the young couple who live next door to her are both under 30 and already have four children. If Keith Stanovich were told about this, what might you reasonably expect him to say?

slide89

A. The census data must be wrong.

B. Anne’s comment illustrates valid

probabilistic reasoning.

C. Anne’s comment illustrates the use of

“person-who” statistics.

D. The young couple provide an exception

that actually serves to prove the rule.

slide90

To learn about the attitudes of Canadian voters toward a government bill limiting auto emissions, Gary gave a questionnaire to 200 members of a group called Friends of a Cleaner Environment. He found that every single one of them strongly supported the bill. Based on this research, which of these is the most appropriate conclusion?

slide91

A. The questionnaire is very reliable.

B. The questionnaire items are not clearly written.

C. The majority of Canadian voters strongly support the bill.

D. The results tell us very little about the attitudes of Canadian voters.

assessing higher level thinking1
Assessing Higher-level Thinking

Although MC tests too often focus on what students can remember, this can usually be avoided by

  • using challenging individual MC items

Ramp it up! (Bloom’s Revised Taxonomy)

Item shells

  • using interpretive exercises
slide93

Item Shells

Haladyna and Shindoll (1989) describe an item shell as a “hollow” MC item that has a syntactic structure, but no content. The test writer can insert important concepts into the item shell to construct challenging MC items.

slide94

Which of these is an example of values found in the Fibonacci series?

  • …34, 55, 89, 144…
  • …49, 57, 65, 73…
  • B. …64, 128, 256, 512…
  • …81, 100, 121, 144…
slide95

Which of these is an example of values found in the Fibonacci series?

  • …34, 55, 89, 144…
  • …49, 57, 65, 73…
  • B. …64, 128, 256, 512…
  • …81, 100, 121, 144…
slide96

Which is a difference between r andρ?

A. r is a parameter,andρ is a statistic.

B. ρ is a parameter,and r is a statistic.

C. r is a continuous variable, and ρ is a

discrete variable.

D. ρis a continuous variable, and ris a

discrete variable.

slide97

Which is a difference betweenr andρ?

A. r is a parameter,andρ is a statistic.

B. ρ is a parameter,and r is a statistic.

C. r is a continuous variable, and ρ is a

discrete variable.

D. ρis a continuous variable, and ris a

discrete variable.

slide98

Which of the following is a major shortcoming of Freud’s theory of personality?

  • Many aspects of the theory cannot be
  • falsified.
  • B. The theory overemphasizes the concept of
  • self-actualization.
  • C. Freud ignores sexuality as a cause of
  • neurotic behaviour.
  • D. Freud did not present his ideas in sufficient
  • detail to permit evaluation of the theory.
slide99

Which of the following is a major shortcoming of Freud’s theory of personality?

  • Many aspects of the theory cannot be
  • falsified.
  • B. The theory overemphasizes the concept of
  • self-actualization.
  • C. Freud ignores sexuality as a cause of
  • neurotic behaviour.
  • D. Freud did not present his ideas in sufficient
  • detail to permit evaluation of the theory.
slide100

Nina wants to reduce the probability of making a Type I error in her research study. Which of these strategies could she use?

A. Increase the sample size.

B. Decrease the value of alpha.

C. Use a one-tailed rather than a two-tailed

test.

D. Use a repeated-measures design rather

than an independent-groups design.

slide101

Nina wants to reduce the probability of making a Type I error in her research study. Which of these strategies could she use?

A. Increase the sample size.

B. Decrease the value of alpha.

C. Use a one-tailed rather than a two-tailed

test.

D. Use a repeated-measures design rather

than an independent-groups design.

assessing higher level thinking2
Assessing Higher-level Thinking

Although MC tests too often focus on what students can remember, this can usually be avoided by

  • using challenging individual MC items

Ramp it up! (Bloom’s Revised Taxonomy)

Item shells

  • using interpretive exercises
slide103
In an interpretive exercise, a number of MC items follow the presentation of novel introductory material, such as a reading, scenario, data set, chart, or map. The MC items can be answered correctly only by referring to the introductory information.
slide104

Tips for Interpretive Exercises

  • Ensure that the introductory material is relevant to the course objectives and appropriate to students’ curricular experience.
  • Ensure that the introductory material is novel, and is as brief and clearly written as possible.
slide105

Tips for Interpretive Exercises

  • Construct MC items that require analysis and interpretation of the introductory material, not simple fact-finding.
  • Let the number of MC items be roughly proportional to the length of introductory material.
slide106

Tips for Interpretive Exercises

  • Interpretive exercises can be used not only with MC items, but also with other response formats, including constructed-response formats.
  • To reuse an interpretive exercise, create new introductory material, and make whatever minimal changes are necessary to the individual MC items.
overview4
Overview
  • About multiple-choice testing
  • Two challenges in MC testing
  • Addressing the challenges

1. Constructing high-quality items

2. Assessing higher-level thinking