item analysis outline l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Item Analysis - Outline PowerPoint Presentation
Download Presentation
Item Analysis - Outline

Loading in 2 Seconds...

play fullscreen
1 / 44

Item Analysis - Outline - PowerPoint PPT Presentation


  • 253 Views
  • Uploaded on

Item Analysis - Outline. 1. Types of test items Selected response items Constructed response items 2. Parts of test items 3. Guidelines for writing test items. Item Analysis - Outline. 4. Item Analysis Distracter measures Item difficulty measures Item discrimination measures

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Item Analysis - Outline' - sheng


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
item analysis outline
Item Analysis - Outline

1. Types of test items

  • Selected response items
  • Constructed response items

2. Parts of test items

3. Guidelines for writing test items

item analysis outline2
Item Analysis - Outline

4. Item Analysis

  • Distracter measures
  • Item difficulty measures
  • Item discrimination measures

5. Item Response Theory

  • ICCS
  • Adaptive testing
1 types of test items
1. Types of test items
  • Selected response
    • Multiple choice
    • Likert scale
    • Category
    • Q-sort
  • Constructed response
a selected response
Multiple choice or forced choice

Task is to choose between set answers

Advantage: ease of scoring

Advantage: scoring requires little skill

Disadvantage: may test memory rather than comprehension

A. Selected response
a selected response5
Multiple choice or forced choice

Correct response must be distinct

Distracters should not be obvious or ambiguous

If distracters are bad,more = less reliable test

Use 3-4 distracters per item

A. Selected response
a selected response6
Multiple choice or forced choice

Likert format

Test-taker chooses a point on a scale that expresses their attitude or belief

Data lend themselves to factor analysis

A. Selected response
likert scale example item
Likert scale example item

Parking costs at the university are fair

1 2 3 4 5

Strongly agree neutral disagree strongly

agree disagree

a selected response8
Multiple choice or forced choice

Likert format

Category

Similar to Likert but with more choices

Test-taker’s commitment

Reliability depends on good instructions & # of categories (≤ 10)

Scoring shows context effects

A. Selected response
a selected response9
Multiple choice or forced choice

Likert format

Category

Q-sort

A large set of cards each with statement referring to a “target”

Test-take sorts cards into piles in terms of how accurate statements are as a description of target

Generally 9 piles

A. Selected response
1 types of test items10
1. Types of test items
  • Selected response
  • Constructed response
    • Free response
    • Fill-in-the-blank
    • Essay tests
    • Portfolios
    • In-basket technique
b constructed response items
Free response

Test-taker responds without constraint

Describes what is important to him/her

B. Constructed response items
b constructed response items12
Free response

Fill-in-the-blank

Used to test for knowledge or to find out about beliefs and attitudes

B. Constructed response items
b constructed response items13
Free response

Fill-in-the-blank

Essay tests

Preferred when you want to assess test-taker’s ability to think analytically, integrate ideas, and express himself

B. Constructed response items
b constructed response items14
Free response

Fill-in-the-blank

Essay tests

Portfolios

Not really a test

Collections of things the person being evaluated has produced

Let you evaluate things you can’t assess with a selected response test

B. Constructed response items
b constructed response items15
Free response

Fill-in-the-blank

Essay tests

Portfolios

In-basket technique

Used in business

Job candidate gets a set of “everyday” problems, says how he or she would deal with those problems

Requires expert raters to grade response

B. Constructed response items
b constructed response items16
Strengths

Assess higher-order skills

More useful feedback to test-taker

Positive influence on study habits?

Easier to create items

B. Constructed response items
b constructed response items17
Weaknesses

Time consuming to use

Possible subjectivity in scoring

B. Constructed response items
2 parts of test items
2. Parts of test items
  • Stimulus or item stem
  • Response format or method
  • Conditions governing the response
  • Procedures for scoring the response
2 parts of test items20
Response format or method

Typically multiple choice or constructed response

2. Parts of test items
2 parts of test items21
Conditions governing the response

e.g., time limits; allowing probes for ambiguous responses; how response is recorded...

2. Parts of test items
2 parts of test items22
Procedures for scoring the response

particularly important for constructed response items

2. Parts of test items
2 parts of test items23
To some extent, your choices on each of these parts will be dictated by:

Precedent

What did you do last time?

Experience

Did that work?

Practical considerations

How many people have to be tested?

How much time is available?

2. Parts of test items
3 writing test items guidelines
3. Writing test items – guidelines
  • Define clearly
  • Generate a pool of potential items
  • Monitor reading level
  • Use unitary items
  • Avoid long items
  • Break any response “set”
3 writing test items guidelines25
Define clearly

Why are you testing?

What do you want to know?

3. Writing test items – guidelines
3 writing test items guidelines26
Define clearly

Generate a pool of potential items

The larger the pool of items you select from, the better the test

Selection from this pool based on item-analysis (see below)

3. Writing test items – guidelines
3 writing test items guidelines27
Define clearly

Generate a pool of potential items

Monitor reading level

level too low?

more sophisticated test-takers may get bored

level too high?

you’re testing reading skill as well as domain you think you’re testing

3. Writing test items – guidelines
3 writing test items guidelines28
Define clearly

Generate a pool of potential items

Monitor reading level

Use unitary items

Then the meaning of the response is clear

3. Writing test items – guidelines
3 writing test items guidelines29
Define clearly

Generate a pool of potential items

Monitor reading level

Use unitary items

Avoid long items

Longer items are more likely to be mis-interpreted by test-takers

Short items are more likely to be unitary

3. Writing test items – guidelines
3 writing test items guidelines30
Define clearly

Generate a pool of potential items

Monitor reading level

Use unitary items

Avoid long items

Break any response “set”

Use reverse-scored items to prevent test-taker’s from getting into a response set such as just responding “5” for every item on a Likert scale

3. Writing test items - guidelines
4 item analysis
4. Item analysis
  • Multiple choice distracter analysis
  • Item difficulty measure P
  • Discrimination index D
  • Item – total correlation
a multiple choice distracter measures
How many people choose each distracter?

Distracters should be equally attractive

Correct choice should be based on knowledge

Where knowledge is lacking, choice should be random

A. Multiple choice – distracter measures
b item difficulty measure p34
P = .50 is best

P = 0 or P = 1 – such items do not distinguish ability levels

B. Item Difficulty Measure P
c item discrimination measures
C. Item Discrimination Measures
  • Discrimination index D
  • Item-total correlation
discrimination index d
Extreme groups method

U = # getting item correct in ‘top’ group

L = # getting item correct in ‘bottom’ group

nU = # in top group

nL = # in bottom group

D = U – L

nU nL

Discrimination Index D
item total correlation
Good item

High correlation

People who get item correct have high score on the test

People who get item wrong have low score on the test

Poor item

Low correlation: look at wording – may be testing reading skill

Item Total Correlation
5 item response theory
5. Item Response theory
  • Item characteristic curves
  • Adaptive testing using computers
a item characteristic curves
Most important idea: Item Characteristic Curves (ICCs)

One curve for each test item

X axis: test-taker ability (given by test score)

Y axis: probability of choosing an answer

A. Item characteristic curves
slide40

Probability of correct response

Item 1

Item 2

Item 3

Test Score

a item characteristic curves41
Slope: how quickly the curve rises.

indicates how well item discriminates among persons of differing abilities

like P(i) in Classical Test Theory

but sample-invariant

A. Item Characteristic Curves
problems with item response theory
Obtaining stable estimates of IRT parameters requires rather large samples

Computationally complex

IRT model assumes that the trait being measured is one-dimensional. It may not be.

Problems with Item Response Theory
b adaptive testing using computers
computer selects harder or easier questions as test-taker gets each question right or wrong

lets you tailor questions for each test-taker

test-taker does not spend most of their time with questions that are too easy or too difficult

B. Adaptive Testing Using Computers