# MEASUREMENT Goal - PowerPoint PPT Presentation

1 / 22

MEASUREMENT Goal. To develop reliable and valid measures using state-of-the-art measurement models Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss. Why Item Response Theory?. Item Response Theory (IRT).

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

MEASUREMENT Goal

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

### MEASUREMENT Goal

• To develop reliable and valid measures using state-of-the-art measurement models

• Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss

### Item Response Theory (IRT)

• A family of mathematical descriptions of what happens when a person meets a test or survey question

• Relates characteristics of items (item parameters) and characteristics of persons (person latent traits) to the probability of a correct or rating/categorical response

• Models the test-taking behavior at the item level

### Item-Person Map

Person Latent Trait

Poor

Good

Q Q Q

Q Q Q

Q Q Q

Q Q Q

Likely

(“easy”)

Unlikely

(“hard”)

Q Q Q

Q Q Q

Q Q Q

Q Q Q

Item Location

Chang & Gehlert (2002).

1-PL (Rasch)

Difficulty (b)

2-PL

Difficulty (b)

Discriminating (a)

3-PL

Difficulty (b)

Discriminating (a)

Guessing (c)

### Polytomous IRT Models

• Polytomous

• 1-PL (threshold)

• Partial Credit

• Rating Scale

• 2-PL (threshold & discriminating)

• Nominal

• Generalized Partial Credit

2=Yes, Limited a little

1=Yes, Limited a lot

3=No, Not Limited at all

1

2

3

* Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports

### Potential Advantages of Using IRT in “Geriatric” Pain Assessment

• Refine existing instruments

• Evaluate item and scale characteristics

• Evaluate different response formats

• Detect differential item functioning

• Evaluate person fit (clinical diagnosis)

• Establish item banks and brief forms

### Item Banking and CAT

A

B

C

D

E

F

new

Item Pool (Sets of Questions)

IRT

Q

Q

Q

Q

Item Bank (Catalogued; Hierarchically Structured)

CAT

Brief Forms

• IRT pre-calibrated item bank

• Initial item selection

• Test scoring method

• Item selection during test administration

• Stopping rules

### Item Bank

• Set of carefully IRT-calibrated questions

• Items covers entire latent trait continuum

• Items represent differing amounts of trait

• Items represent differing amounts of information

• Items can be selected to maximize precision and retain clinical relevance

### Item Banking is Inter-disciplinary

• Psychometricians

• Information scientists

• Clinicians/healthcare providers

• Outcomes researchers

• Content experts

### Approaches to Develop Item Banks

• Top-Down Approach

• Bottom-Up Approach

### Development and Maintenance of an Item Bank

• How to best calibrate existing items?

• Model selection

• Whose item parameters to use?

• Standardization?

• Generic vs. disease-specific

• Item parameter drift

• Anchor or Re-calibrate?

• How to write and best test new items?

• An adaptive test is a tailored, individualized measure which involves selecting a set of test items for each individual that best measures the psychological characteristics of that person (Weiss, 1985)

• Weiss DJ. Adaptive testing by computer. J Consult Clin Psychol. Dec 1985;53(6):774-789.

• Adaptive testing selects questions based on previous responses

• Tailored item and test difficulties

• Eliminates floor and ceiling effects

• Require fewer questions to arrive at an accurate estimate

• Automate question administration, data recording, scoring, and prompt reporting

• Allows for immediate feedback

### CAT Algorithm

Administer Item of Median Difficulty (or Screening Item)

Score Item

Estimate Latent Trait (Theta)

Termination Criterion Satisfied

Choose and Administer Next Item with Maximum Information

No

Yes

Stop

### Increase of Accuracy of Ability or Latent Trait Estimation in CAT

For each item added to the test, the width of the interval decreases.

Item 1-5

Item 1-4

Item 1-3

Item 1-2

Item 1

Ability ()

### Potential Problems with CAT in Pain and Health Outcomes Measurement

• Context effects

• Unbalanced content

• Time frame

• Response categories

• Multidimensionality

### What kind of short form?

• Question 1

• 0 I do not feel sad.

• 2 I am sad all the time and I can’t snap out of it.

• 3 I am so sad or unhappy that I can’t stand it.

Are you basically satisfied with your life?True/False

Item production

Item statistics

Item exposure

Maintaining a valid bank of items for test construction

Fairness

Delivery options

Cost-benefit considerations

### Infrastructure of a National Geriatric Pain Item Bank

Subscriber

Public

Individual Researchers

Pharm. Industries

Non-profit Institutions

Government Agencies

Customized Information Retrieval; CAT; (automated) Brief Form

National “Central” Item Bank

Collector

Analyzer

Builder

Retriever

Consortium Approval

IRT Analyses

Item Parameters

### An Integrated Solution for Pain and Outcomes Assessments

Chang, C.-H., & Yang, D. (2003, April 15). Patient-Reported Outcomes Information Technology: The PROsITTM System. ISPOR CONNECTIONS, 9(2), 5-6.