Basic Testing Issues

Presentation Transcript

2007 Annual Conference

Ron Rodgers

CTS/Employment Research Institute

Session Agenda

A. Clarify test development Standards

Summarize test development process
Write & validate multiple choice items
Respond to your questions

Council on Licensure, Enforcement and Regulation

APA Standards*(1954-1999)

APA Standards used by US Supreme Court to validate fair, valid examinations
Test must reflect job qualifications
Credentialing identifies qualified practitioners for public, employers
Panels of experts should help specify certification performance standards

* Standards for Educational and Psychological Testing, American Psychological Association, American Educational Research Association, National Council on Measurement in Education

APA Standards*(1954-1999)

Candidates who fail receive report w/information about weak areas
Credentialing must depend on the knowledge and skills needed for acceptable professional performance
Criteria should not be adjusted to regulate the number or proportion of persons who pass the test

Define Exam Objectives

Create SME panel to represent experienced professionals in field
Identify references & initial specs
Conduct job analysis
Link test specifications to job tasks

Write & Validate Questions

Select item writing model
Draft item bank to meet test specs
Document each Q to references
Validate/edit Qs with SME panel

Score & Report Results

Assemble test forms
Set passing criteria
Review item performance statistics
Equate test forms
Adjust scoring if needed

Provide Candidate Info & Tech Support

Prepare candidate materials
Monitor test & item performance
Revalidate non-performing Qs
Amend items/test forms as needed

Maintain Test Security

Limit access to secure materials
Use non-disclosure agreements
Emphasize security to candidates
Define consequences of cheating

Non-Disclosure Agreement

Each Subject Matter Expert (SME) acknowledges that all work performed under this Agreement for and on behalf of (program sponsor) is confidential and that no disclosure of any aspects of test data, information, materials and questions is permitted. The SME also agrees to adhere to all test security procedures and requirements specified by __________.

Responding to Your Questions

Writing good test questions

Bruce Anderson, PES

Identifying your concerns

Bruce Anderson, PES

Ron Rodgers, CTS

Finding sources of assistance

Speaker Contact Information

Ron Rodgers

Continental Testing Service

Employment Research Institute

809 Ridge Road, Suite 201

Wilmette, IL 60091

Phone: 847-256-5240

Email: rrodgers@Qmail.com

2007 Annual Conference

“Writing Good Multiple-Choice Items”

Bruce Anderson

Professional Examination Service

Agenda

Overview of the examination development process
Review of item-writing principles
Filling out the item-writing form
Review and development of items

Developing the Certification Examination

Conduct a job analysis/role delineation.
Generate test specifications.
Write items.
Validate/Improve items.
Enter items into item bank.
Assemble draft examinations.
Review and revise examination forms.

Developing the Certification Examination (contd.)

Conduct a passing point study.
Review item statistics.
Equate examination forms.
Score and report examination results.

What are the chances of an uninformed candidate selecting the correct answer to your item?

1 in 4, and that’s if you write a good item!!

Guessing the Correct Answer

For a multiple-choice item with four possible responses:
3 good distracters = a 25% chance
1 BAD distracter = a 33% chance
2 BAD distracters = a 50% chance
3 BAD distracters = a 100% chance

of an uninformed candidate guessing the correct answer.

The Effects of Guessing on the Value of the Certification

The greater the ability of the uninformed candidate to guess correctly, the less valid and reliable the examination is.
The less valid and reliable the examination is, the less value is placed on the certification.

The Critical Objective when Writing Good Multiple-Choice Items

Reducing the ability of uninformed candidates to guess correctly by providing NO CLUES to the correct answer.

Where are the clues which help candidates to guess correctly?

The type of format chosen for the question
The format of the question asked (STEM)
The way the correct answer (KEY) is written
The way the incorrect answers (DISTRACTORS) are written

The Format of the Stem

DIRECT QUESTION:

Which of the following scientists was responsible for the development of the first polio vaccine?

or

INCOMPLETE STATEMENT:

The first polio vaccine was developed by:

Jonas Salk
Louis Pasteur
Edward Jenner
Robert Koch

What is the capital of Kentucky?

Frankfort*

Lexington

Louisville

Ashland

The major goal of raising the legal drinking age is to:

Reduce driving fatalities and injuries.*

Reduce liquor consumption.

Discourage the habit of drinking by adolescents.

Reduce roadside litter.

Correct vs. Best Answer Stems

Guidelines for Constructing the Stem

Clarity and Directness: Is the requirement of the item clear without reference to the choices?
Sufficiency: Is enough information provided to answer the question?
Focus: Is the item focused on a single problem which assesses a meaningful fact or concept?

EXAMPLE: Clarity? Sufficiency? Focus?

The 2000 U.S. Presidential Election was decided by:
1. fewer than 1000 votes
2. the Electoral College*
3. the Supreme Court
4. December 21, 2000

Guidelines for Constructing the Stem (cont’d)

Avoid negative stems.
Provide just enough information.
Do not use a word in the stem which only appears in the choice which is the key.
Do not provide gratuitous instruction.
Provide most information in the stem so that the options do not have repetition of words.

EXAMPLE: Negatives are confusing.

Which of the following authors is not a major American novelist?
1. George Orwell
2. William Faulkner
3. John Steinbeck
4. Ernest Hemmingway

EXAMPLE: Improved

Which of the following authors is a major British novelist?
1. George Orwell
2. William Faulkner
3. John Steinbeck
4. Ernest Hemmingway

Poor

In 1850, Adolphe Chatin, the professor of pharmacy in Paris, believed that goiter resulted from an inadequate amount of iodine in the diet. The thyroid is part of which body system?

Endocrine*

Nervous

Musculoskeletal

Cardiovascular

Better

The thyroid is part of which body system?

1. Endocrine*

2. Nervous

3. Musculoskeletal

4. Cardiovascular

EXAMPLE: Gratuitous instruction

Guidelines for the Correct Answer (KEY)

Is the intended key unambiguously correct and recognized as such by the experts?
Is the key parallel to the other choices (e.g., not shorter or longer, or more technical, or more detailed)?
Is there ONLY ONE clearly correct or best answer?

Guidelines for Distracters

SOURCES OF DISTRACTORS

Common misconceptions
Common job performance errors
Statements which are true but which do not satisfy the requirements of the question
Statements which are either too broad or too narrow for the requirements of the question
Carefully worded incorrect statements which may sound plausible to the uninformed

Guidelines for The Choices

Does each choice follow logically and grammatically from the stem?
Are the choices parallel in:
length?
grammatical structure?
terminology?
content?
level of detail?

An American city noted for its production of automobiles is:

1. Detroit*

2. Ohio

3. Indiana

4. Philadelphia

An American city noted for its production of automobiles is:

Detroit*

Cincinnati

Indianapolis

Philadelphia

EXAMPLES: Parallelism

Serious communicable diseases that can cause developmental disability to the offspring of a pregnant mother are:

1. rubella and chicken pox*

2. cancer

3. rheumatism and arthritis

4. sinus infection

A serious communicable disease that can cause developmental disability to the offspring of a pregnant mother is:

rubella*

cancer

rheumatism

sinus infection

EXAMPLES: Parallelism, length.

Guidelines for Choices (contd.)

Do not repeat the same word in each answer choice; include the word once in the stem. As much time as possible is to be spent figuring out the answer rather than reading.
Make the choices mutually exclusive.
Do not use “all of the above” or “none of the above” as choices. The latter, for example, does not test whether a candidate knows the correct answer.

The right to vote in the United States is granted to individuals of what age?

1. 16

2. 17

3. 18*

4. 19

The right to vote in the United States is granted to individuals beginning at what age?

1. 16

2. 17

3. 18*

4. 19

EXAMPLES: Overlap

General Considerations

Is it RELEVANT to professional practice?
Is current BEST practice in the field tested?
Does it test an important concept of piece of knowledge?
Is it appropriate for entry-level?
Is the answer clear to a prepared candidate (not a ‘trick question’)?

General Considerations

Does it avoid controversial issues, e.g., religious, political, philosophical?
Are the context, setting, and content equally appropriate and familiar to all segments of the candidate population?
Is the terminology equally appropriate and familiar to all segments of the candidate population (e.g., international candidates)?
Is it free of bias re: persons or groups?

EXAMPLE: Sensitive Issue

Of the following profile analysis groups, the one most likely to have alcohol-related problems is:
1. residents of rural areas in the South
2. widowed males over 60 years of age
3. women who are married and are wine drinkers
4. men with no religious affiliation*

Other Sensitivity Issues

Avoid humor in test questions.
Avoid any subject that appears to be advancing a political or business agenda.
Avoid topics that could cause undue stress to the candidates, e.g., death, disease, war, brutality.
Avoid any stereotypes.

Rules of English

Don’t use no double negatives.
Make each pronoun agree with their antecedent.
Join clauses good, like a conjunction should.
Watch them participles when dangling.
About them sentence fragments.
Verbs has to agree with their subject.
Just between you and I, case is important too.
Don’t write run-on sentences they are hard to read.
Don’t use commas, that are not necessary.
Do’nt misuse apostrophes; its’ wrong.

THE I GNU TEST

1. The purpose of the cluss in furmpaling is to remove:

1. cluss-prags
2. tremalis
3. cloughs
4. plumots

2. Trassig is true when:

1. lusp trasses the vom
2. the viskal flans, if the viskal is donwill or zortil
3. the belgo frulls
4. dissles lisk easily

3. The sigla frequently overfesks the treslum because:

1. all siglas are mellious
2. siglas are always votial
3. the treslum is usually tarious
4. no trelsa are feskable

4. The fribbled breg will minter best with an:

1. derst
2. morst
3. sortar
4. ignu

5. The mintering function of the ignu is most effectively carried out in connection with:

a razma tol
the groshing stantol
the fribbled breg
a frally sush

Problem Items

Which of the following persons was President of the United States when the Civil Rights Act of 1964 was enacted?
A. Lester Pearson
B. Anthony Eden
C. Jefferson Davis
D. Lyndon Johnson