basic testing issues l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Basic Testing Issues PowerPoint Presentation
Download Presentation
Basic Testing Issues

Loading in 2 Seconds...

play fullscreen
1 / 52

Basic Testing Issues - PowerPoint PPT Presentation


  • 151 Views
  • Uploaded on

2007 Annual Conference. Basic Testing Issues. Ron Rodgers CTS/Employment Research Institute. Session Agenda. A. Clarify test development Standards Summarize test development process Write & validate multiple choice items Respond to your questions. APA Standards* (1954-1999).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Basic Testing Issues' - fifi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
basic testing issues

2007 Annual Conference

Basic Testing Issues

Ron Rodgers

CTS/Employment Research Institute

session agenda
Session Agenda

A. Clarify test development Standards

  • Summarize test development process
  • Write & validate multiple choice items
  • Respond to your questions

Council on Licensure, Enforcement and Regulation

apa standards 1954 1999
APA Standards*(1954-1999)
  • APA Standards used by US Supreme Court to validate fair, valid examinations
  • Test must reflect job qualifications
  • Credentialing identifies qualified practitioners for public, employers
  • Panels of experts should help specify certification performance standards

* Standards for Educational and Psychological Testing, American Psychological Association, American Educational Research Association, National Council on Measurement in Education

Council on Licensure, Enforcement and Regulation

apa standards 1954 19994
APA Standards*(1954-1999)
  • Candidates who fail receive report w/information about weak areas
  • Credentialing must depend on the knowledge and skills needed for acceptable professional performance
  • Criteria should not be adjusted to regulate the number or proportion of persons who pass the test

Council on Licensure, Enforcement and Regulation

define exam objectives
Define Exam Objectives
  • Create SME panel to represent experienced professionals in field
  • Identify references & initial specs
  • Conduct job analysis
  • Link test specifications to job tasks

Council on Licensure, Enforcement and Regulation

write validate questions
Write & Validate Questions
  • Select item writing model
  • Draft item bank to meet test specs
  • Document each Q to references
  • Validate/edit Qs with SME panel

Council on Licensure, Enforcement and Regulation

score report results
Score & Report Results
  • Assemble test forms
  • Set passing criteria
  • Review item performance statistics
  • Equate test forms
  • Adjust scoring if needed

Council on Licensure, Enforcement and Regulation

provide candidate info tech support
Provide Candidate Info & Tech Support
  • Prepare candidate materials
  • Monitor test & item performance
  • Revalidate non-performing Qs
  • Amend items/test forms as needed

Council on Licensure, Enforcement and Regulation

maintain test security
Maintain Test Security
  • Limit access to secure materials
  • Use non-disclosure agreements
  • Emphasize security to candidates
  • Define consequences of cheating

Council on Licensure, Enforcement and Regulation

non disclosure agreement
Non-Disclosure Agreement

Each Subject Matter Expert (SME) acknowledges that all work performed under this Agreement for and on behalf of (program sponsor) is confidential and that no disclosure of any aspects of test data, information, materials and questions is permitted. The SME also agrees to adhere to all test security procedures and requirements specified by __________.

Council on Licensure, Enforcement and Regulation

responding to your questions
Responding to Your Questions
  • Writing good test questions

Bruce Anderson, PES

  • Identifying your concerns

Bruce Anderson, PES

Ron Rodgers, CTS

  • Finding sources of assistance

Council on Licensure, Enforcement and Regulation

speaker contact information
Speaker Contact Information

Ron Rodgers

Continental Testing Service

Employment Research Institute

809 Ridge Road, Suite 201

Wilmette, IL 60091

Phone: 847-256-5240

Email: rrodgers@Qmail.com

Council on Licensure, Enforcement and Regulation

writing good multiple choice items

2007 Annual Conference

“Writing Good Multiple-Choice Items”

Bruce Anderson

Professional Examination Service

agenda
Agenda
  • Overview of the examination development process
  • Review of item-writing principles
  • Filling out the item-writing form
  • Review and development of items

Council on Licensure, Enforcement and Regulation

developing the certification examination
Developing the Certification Examination
  • Conduct a job analysis/role delineation.
  • Generate test specifications.
  • Write items.
  • Validate/Improve items.
  • Enter items into item bank.
  • Assemble draft examinations.
  • Review and revise examination forms.

Council on Licensure, Enforcement and Regulation

developing the certification examination contd
Developing the Certification Examination (contd.)
  • Conduct a passing point study.
  • Review item statistics.
  • Equate examination forms.
  • Score and report examination results.

Council on Licensure, Enforcement and Regulation

what are the chances of an uninformed candidate selecting the correct answer to your item
What are the chances of an uninformed candidate selecting the correct answer to your item?

1 in 4, and that’s if you write a good item!!

Council on Licensure, Enforcement and Regulation

guessing the correct answer
Guessing the Correct Answer
  • For a multiple-choice item with four possible responses:
  • 3 good distracters = a 25% chance
  • 1 BAD distracter = a 33% chance
  • 2 BAD distracters = a 50% chance
  • 3 BAD distracters = a 100% chance

of an uninformed candidate guessing the correct answer.

Council on Licensure, Enforcement and Regulation

the effects of guessing on the value of the certification
The Effects of Guessing on the Value of the Certification
  • The greater the ability of the uninformed candidate to guess correctly, the less valid and reliable the examination is.
  • The less valid and reliable the examination is, the less value is placed on the certification.

Council on Licensure, Enforcement and Regulation

the critical objective when writing good multiple choice items
The Critical Objective when Writing Good Multiple-Choice Items
  • Reducing the ability of uninformed candidates to guess correctly by providing NO CLUES to the correct answer.

Council on Licensure, Enforcement and Regulation

where are the clues which help candidates to guess correctly
Where are the clues which help candidates to guess correctly?
  • The type of format chosen for the question
  • The format of the question asked (STEM)
  • The way the correct answer (KEY) is written
  • The way the incorrect answers (DISTRACTORS) are written

Council on Licensure, Enforcement and Regulation

the format of the stem
The Format of the Stem

DIRECT QUESTION:

Which of the following scientists was responsible for the development of the first polio vaccine?

or

INCOMPLETE STATEMENT:

The first polio vaccine was developed by:

  • Jonas Salk
  • Louis Pasteur
  • Edward Jenner
  • Robert Koch

Council on Licensure, Enforcement and Regulation

correct vs best answer stems
What is the capital of Kentucky?

Frankfort*

Lexington

Louisville

Ashland

The major goal of raising the legal drinking age is to:

Reduce driving fatalities and injuries.*

Reduce liquor consumption.

Discourage the habit of drinking by adolescents.

Reduce roadside litter.

Correct vs. Best Answer Stems

Council on Licensure, Enforcement and Regulation

guidelines for constructing the stem
Guidelines for Constructing the Stem
  • Clarity and Directness: Is the requirement of the item clear without reference to the choices?
  • Sufficiency: Is enough information provided to answer the question?
  • Focus: Is the item focused on a single problem which assesses a meaningful fact or concept?

Council on Licensure, Enforcement and Regulation

example clarity sufficiency focus
EXAMPLE: Clarity? Sufficiency? Focus?
  • The 2000 U.S. Presidential Election was decided by:
  • 1. fewer than 1000 votes
  • 2. the Electoral College*
  • 3. the Supreme Court
  • 4. December 21, 2000

Council on Licensure, Enforcement and Regulation

guidelines for constructing the stem cont d
Guidelines for Constructing the Stem (cont’d)
  • Avoid negative stems.
  • Provide just enough information.
  • Do not use a word in the stem which only appears in the choice which is the key.
  • Do not provide gratuitous instruction.
  • Provide most information in the stem so that the options do not have repetition of words.

Council on Licensure, Enforcement and Regulation

example negatives are confusing
EXAMPLE: Negatives are confusing.
  • Which of the following authors is not a major American novelist?
  • 1. George Orwell
  • 2. William Faulkner
  • 3. John Steinbeck
  • 4. Ernest Hemmingway

Council on Licensure, Enforcement and Regulation

example improved
EXAMPLE: Improved
  • Which of the following authors is a major British novelist?
  • 1. George Orwell
  • 2. William Faulkner
  • 3. John Steinbeck
  • 4. Ernest Hemmingway

Council on Licensure, Enforcement and Regulation

example gratuitous instruction
Poor

In 1850, Adolphe Chatin, the professor of pharmacy in Paris, believed that goiter resulted from an inadequate amount of iodine in the diet. The thyroid is part of which body system?

Endocrine*

Nervous

Musculoskeletal

Cardiovascular

Better

The thyroid is part of which body system?

1. Endocrine*

2. Nervous

3. Musculoskeletal

4. Cardiovascular

EXAMPLE: Gratuitous instruction

Council on Licensure, Enforcement and Regulation

guidelines for the correct answer key
Guidelines for the Correct Answer (KEY)
  • Is the intended key unambiguously correct and recognized as such by the experts?
  • Is the key parallel to the other choices (e.g., not shorter or longer, or more technical, or more detailed)?
  • Is there ONLY ONE clearly correct or best answer?

Council on Licensure, Enforcement and Regulation

guidelines for distracters
Guidelines for Distracters

SOURCES OF DISTRACTORS

  • Common misconceptions
  • Common job performance errors
  • Statements which are true but which do not satisfy the requirements of the question
  • Statements which are either too broad or too narrow for the requirements of the question
  • Carefully worded incorrect statements which may sound plausible to the uninformed

Council on Licensure, Enforcement and Regulation

guidelines for the choices
Guidelines for The Choices
  • Does each choice follow logically and grammatically from the stem?
  • Are the choices parallel in:
  • length?
  • grammatical structure?
  • terminology?
  • content?
  • level of detail?

Council on Licensure, Enforcement and Regulation

examples parallelism
An American city noted for its production of automobiles is:

1. Detroit*

2. Ohio

3. Indiana

4. Philadelphia

An American city noted for its production of automobiles is:

Detroit*

Cincinnati

Indianapolis

Philadelphia

EXAMPLES: Parallelism

Council on Licensure, Enforcement and Regulation

examples parallelism length
Serious communicable diseases that can cause developmental disability to the offspring of a pregnant mother are:

1. rubella and chicken pox*

2. cancer

3. rheumatism and arthritis

4. sinus infection

A serious communicable disease that can cause developmental disability to the offspring of a pregnant mother is:

rubella*

cancer

rheumatism

sinus infection

EXAMPLES: Parallelism, length.

Council on Licensure, Enforcement and Regulation

guidelines for choices contd
Guidelines for Choices (contd.)
  • Do not repeat the same word in each answer choice; include the word once in the stem. As much time as possible is to be spent figuring out the answer rather than reading.
  • Make the choices mutually exclusive.
  • Do not use “all of the above” or “none of the above” as choices. The latter, for example, does not test whether a candidate knows the correct answer.

Council on Licensure, Enforcement and Regulation

examples overlap
The right to vote in the United States is granted to individuals of what age?

1. 16

2. 17

3. 18*

4. 19

The right to vote in the United States is granted to individuals beginning at what age?

1. 16

2. 17

3. 18*

4. 19

EXAMPLES: Overlap

Council on Licensure, Enforcement and Regulation

general considerations
General Considerations
  • Is it RELEVANT to professional practice?
  • Is current BEST practice in the field tested?
  • Does it test an important concept of piece of knowledge?
  • Is it appropriate for entry-level?
  • Is the answer clear to a prepared candidate (not a ‘trick question’)?

Council on Licensure, Enforcement and Regulation

general considerations38
General Considerations
  • Does it avoid controversial issues, e.g., religious, political, philosophical?
  • Are the context, setting, and content equally appropriate and familiar to all segments of the candidate population?
  • Is the terminology equally appropriate and familiar to all segments of the candidate population (e.g., international candidates)?
  • Is it free of bias re: persons or groups?

Council on Licensure, Enforcement and Regulation

example sensitive issue
EXAMPLE: Sensitive Issue
  • Of the following profile analysis groups, the one most likely to have alcohol-related problems is:
  • 1. residents of rural areas in the South
  • 2. widowed males over 60 years of age
  • 3. women who are married and are wine drinkers
  • 4. men with no religious affiliation*

Council on Licensure, Enforcement and Regulation

other sensitivity issues
Other Sensitivity Issues
  • Avoid humor in test questions.
  • Avoid any subject that appears to be advancing a political or business agenda.
  • Avoid topics that could cause undue stress to the candidates, e.g., death, disease, war, brutality.
  • Avoid any stereotypes.

Council on Licensure, Enforcement and Regulation

rules of english
Rules of English
  • Don’t use no double negatives.
  • Make each pronoun agree with their antecedent.
  • Join clauses good, like a conjunction should.
  • Watch them participles when dangling.
  • About them sentence fragments.
  • Verbs has to agree with their subject.
  • Just between you and I, case is important too.
  • Don’t write run-on sentences they are hard to read.
  • Don’t use commas, that are not necessary.
  • Do’nt misuse apostrophes; its’ wrong.

Council on Licensure, Enforcement and Regulation

the i gnu test
THE I GNU TEST

Council on Licensure, Enforcement and Regulation

1 the purpose of the cluss in furmpaling is to remove
1. The purpose of the cluss in furmpaling is to remove:
  • 1. cluss-prags
  • 2. tremalis
  • 3. cloughs
  • 4. plumots

Council on Licensure, Enforcement and Regulation

2 trassig is true when
2. Trassig is true when:
  • 1. lusp trasses the vom
  • 2. the viskal flans, if the viskal is donwill or zortil
  • 3. the belgo frulls
  • 4. dissles lisk easily

Council on Licensure, Enforcement and Regulation

3 the sigla frequently overfesks the treslum because
3. The sigla frequently overfesks the treslum because:
  • 1. all siglas are mellious
  • 2. siglas are always votial
  • 3. the treslum is usually tarious
  • 4. no trelsa are feskable

Council on Licensure, Enforcement and Regulation

4 the fribbled breg will minter best with an
4. The fribbled breg will minter best with an:
  • 1. derst
  • 2. morst
  • 3. sortar
  • 4. ignu

Council on Licensure, Enforcement and Regulation

5 the mintering function of the ignu is most effectively carried out in connection with
5. The mintering function of the ignu is most effectively carried out in connection with:
  • a razma tol
  • the groshing stantol
  • the fribbled breg
  • a frally sush

Council on Licensure, Enforcement and Regulation

problem items
Problem Items

Council on Licensure, Enforcement and Regulation

problem items49
Problem Items
  • Which of the following persons was President of the United States when the Civil Rights Act of 1964 was enacted?
  • A. Lester Pearson
  • B. Anthony Eden
  • C. Jefferson Davis
  • D. Lyndon Johnson

Council on Licensure, Enforcement and Regulation

problem items50
Problem Items
  • Who orchestrated the December 7, 1941 attack on Pearl Harbor?

A. Isoroku Yamamoto

B. Chester Nimitz

C. Bernard Montgomery

D. Seiji Ozawa

Council on Licensure, Enforcement and Regulation

problem items51
Problem Items
  • Which Swedish opera singer was known for her role in Tristan und Isolde?

A. Liv Ullman

B. Birgit Nilsson

C. Jussi Bjorling

D. Ingmar Bergman

Council on Licensure, Enforcement and Regulation

contact information
Contact Information

Bruce R. Anderson

Senior Program Director

Professional Examination Service

475 Riverside Drive

New York, NY 10115

Phone: 212.367.4239

Fax:  212.367.4396

anderson@proexam.org

Council on Licensure, Enforcement and Regulation