Measurement goal
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

MEASUREMENT Goal PowerPoint PPT Presentation


  • 52 Views
  • Uploaded on
  • Presentation posted in: General

MEASUREMENT Goal. To develop reliable and valid measures using state-of-the-art measurement models Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss. Why Item Response Theory?. Item Response Theory (IRT).

Download Presentation

MEASUREMENT Goal

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Measurement goal

MEASUREMENT Goal

  • To develop reliable and valid measures using state-of-the-art measurement models

    • Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss


Why item response theory

Why Item Response Theory?


Item response theory irt

Item Response Theory (IRT)

  • A family of mathematical descriptions of what happens when a person meets a test or survey question

  • Relates characteristics of items (item parameters) and characteristics of persons (person latent traits) to the probability of a correct or rating/categorical response

  • Models the test-taking behavior at the item level


Item person map

Item-Person Map

Person Latent Trait

Poor

Good

Q Q Q

Q Q Q

Q Q Q

Q Q Q

Likely

(“easy”)

Unlikely

(“hard”)

Q Q Q

Q Q Q

Q Q Q

Q Q Q

Item Location

Chang & Gehlert (2002).


Dichotomous unidimensional irt models

1-PL (Rasch)

Difficulty (b)

2-PL

Difficulty (b)

Discriminating (a)

3-PL

Difficulty (b)

Discriminating (a)

Guessing (c)

Dichotomous Unidimensional IRT Models


Polytomous irt models

Polytomous IRT Models

  • Polytomous

    • 1-PL (threshold)

      • Partial Credit

      • Rating Scale

    • 2-PL (threshold & discriminating)

      • Nominal

      • Graded Response

      • Generalized Partial Credit

2=Yes, Limited a little

1=Yes, Limited a lot

3=No, Not Limited at all

1

2

3

* Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports


Potential advantages of using irt in geriatric pain assessment

Potential Advantages of Using IRT in “Geriatric” Pain Assessment

  • Refine existing instruments

  • Evaluate item and scale characteristics

  • Evaluate different response formats

  • Detect differential item functioning

  • Evaluate person fit (clinical diagnosis)

  • Equate/Link instruments

  • Establish item banks and brief forms

  • Develop computerized adaptive testing


Item banking and cat

Item Banking and CAT

A

B

C

D

E

F

new

Item Pool (Sets of Questions)

IRT

Q

Q

Q

Q

Item Bank (Catalogued; Hierarchically Structured)

CAT

Brief Forms


Principles of adaptive testing

Principles of Adaptive Testing

  • IRT pre-calibrated item bank

  • Initial item selection

  • Test scoring method

  • Item selection during test administration

  • Stopping rules


Item bank

Item Bank

  • Set of carefully IRT-calibrated questions

  • Items covers entire latent trait continuum

  • Items represent differing amounts of trait

  • Items represent differing amounts of information

  • Basis for tailored/adaptive testing

  • Items can be selected to maximize precision and retain clinical relevance


Item banking is inter disciplinary

Item Banking is Inter-disciplinary

  • Psychometricians

  • Information scientists

  • Clinicians/healthcare providers

  • Outcomes researchers

  • Content experts


Approaches to develop item banks

Approaches to Develop Item Banks

  • Top-Down Approach

  • Bottom-Up Approach


Development and maintenance of an item bank

Development and Maintenance of an Item Bank

  • How to best calibrate existing items?

    • Model selection

    • Whose item parameters to use?

    • Standardization?

    • Generic vs. disease-specific

  • Item parameter drift

    • Anchor or Re-calibrate?

  • How to write and best test new items?


Adaptive test

Adaptive Test

  • An adaptive test is a tailored, individualized measure which involves selecting a set of test items for each individual that best measures the psychological characteristics of that person (Weiss, 1985)

  • Weiss DJ. Adaptive testing by computer. J Consult Clin Psychol. Dec 1985;53(6):774-789.


Why computerized adaptive testing

Why Computerized Adaptive Testing?

  • Adaptive testing selects questions based on previous responses

  • Tailored item and test difficulties

  • Eliminates floor and ceiling effects

  • Require fewer questions to arrive at an accurate estimate

  • Automate question administration, data recording, scoring, and prompt reporting

  • Allows for immediate feedback


Cat algorithm

CAT Algorithm

Administer Item of Median Difficulty (or Screening Item)

Score Item

Estimate Latent Trait (Theta)

Termination Criterion Satisfied

Choose and Administer Next Item with Maximum Information

No

Yes

Stop


Increase of accuracy of ability or latent trait estimation in cat

Increase of Accuracy of Ability or Latent Trait Estimation in CAT

For each item added to the test, the width of the interval decreases.

Item 1-5

Item 1-4

Item 1-3

Item 1-2

Item 1

Ability ()


Potential problems with cat in pain and health outcomes measurement

Potential Problems with CAT in Pain and Health Outcomes Measurement

  • Context effects

  • Unbalanced content

  • Time frame

  • Response categories

  • Multidimensionality


What kind of short form

What kind of short form?

  • Question 1

    • 0 I do not feel sad.

    • 1 I feel sad

    • 2 I am sad all the time and I can’t snap out of it.

    • 3 I am so sad or unhappy that I can’t stand it.

Are you basically satisfied with your life?True/False


More research still needed for effective cat implementation

Item production

Item statistics

Item exposure

Maintaining a valid bank of items for test construction

Fairness

Delivery options

Effects of modes of administration

Cost-benefit considerations

MORE Research Still Needed for Effective CAT Implementation


Infrastructure of a national geriatric pain item bank

Infrastructure of a National Geriatric Pain Item Bank

Subscriber

Public

Individual Researchers

Pharm. Industries

Non-profit Institutions

Government Agencies

Customized Information Retrieval; CAT; (automated) Brief Form

National “Central” Item Bank

Collector

Analyzer

Builder

Retriever

Consortium Approval

IRT Analyses

Item Parameters


An integrated solution for pain and outcomes assessments

An Integrated Solution for Pain and Outcomes Assessments

Chang, C.-H., & Yang, D. (2003, April 15). Patient-Reported Outcomes Information Technology: The PROsITTM System. ISPOR CONNECTIONS, 9(2), 5-6.


  • Login