The impact of vertical scaling decisions on growth projections a two act play
Download
1 / 18

The Impact of Vertical Scaling Decisions on Growth Projections: A Two Act Play - PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on

The Impact of Vertical Scaling Decisions on Growth Projections: A Two Act Play. Derek Briggs Jonathan Weeks Ed Wiley University of Colorado, Boulder. Presentation at the annual meeting of the National Council of Measurement in Education. March 26, 2008. New York, NY. Curtains Up.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' The Impact of Vertical Scaling Decisions on Growth Projections: A Two Act Play' - hoyt-chan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The impact of vertical scaling decisions on growth projections a two act play

The Impact of Vertical Scaling Decisions on Growth Projections: A Two Act Play

Derek Briggs

Jonathan Weeks

Ed Wiley

University of Colorado, Boulder

Presentation at the annual meeting of the National Council of Measurement in Education. March 26, 2008. New York, NY.


Curtains up
Curtains Up Projections: A Two Act Play

PLOT SYNOPSIS

  • Growth models require some form of longitudinal data.

  • Implicit assumption that test scores have a consistent interpretation over time.

  • There are multiple technical decisions to make when creating a vertical score scale.

  • Do these decisions have a sizeable impact on growth projections?


ACT I: Projections: A Two Act Play

Creating Vertical Scales


Data Projections: A Two Act Play

  • Outcome measure are Colorado Student Asessment Program (CSAP) test scores in reading. [Items: ~70 MC, 14 CR]

  • Longitudinal item responses for two cohorts of public and charter school students in the state of Colorado.

  • Each grade by year cell combination contains roughly 56,000 students. 1,379 unique schools.

  • Race/Ethnicity of Students: 64% White, 26% Hispanic, 6.3% Black


Linking design
Linking Design Projections: A Two Act Play

  • (MC items, CR items) Unique Items

  • (MC items, CR items) Common Items


Creating a vertical scale
Creating a Vertical Scale Projections: A Two Act Play

Technical Decisions Psychometricians Make

  •  Estimation

  • EAP

  • ML

  • IRT Model

  • 1PLM/PCM

  • 3PLM/GPCM

  • Calibration

  • Separate

  • Concurrent

8 Defensible Vertical Scales

  • se1

  • sm1

  • se3

  • sm3

  • ce1

  • cm1

  • ce3

  • cm3


Separate calibration
Separate Calibration Projections: A Two Act Play

Each oval represents a the linking of two separate item calibrations using the Stocking & Lord ICC approach.


Hybrid calibration
Hybrid Calibration Projections: A Two Act Play

  • Each oval represents a the linking of two separate item calibrations using the Stocking & Lord ICC approach.

  • Each rectangle represents the concurrent, multigroup calibration of the same grade level across two years.


ACT 2: Projections: A Two Act Play

Projections on Vertical Scales


Growth model
Growth Model Projections: A Two Act Play

  • Linear Mixed Effects Model (3 Level HLM)

  • Given 3 years of test score data for a student (grades 3-5), project a scale score 3 years later (grade 8).

  • Score projection is a function of

    • two fixed effects (intercept & slope)

    • two student level random effects (level 2 intercept & slope)

    • two school level random effects (level 3 intercept & slope)


Fixed effect estimates
Fixed Effect Estimates Projections: A Two Act Play

Note: Scale Score Outcome is in Logit Units, Base Year = Grade 3


Comparing growth projections
Comparing Growth Projections Projections: A Two Act Play

3PLM/GPCM &Separate

3PLM/GPCM & Hybrid

1PLM/PCM

Note: Projection lines based solely on fixed effect estimates

Grade 3

Grade 8




Empircal bayes estimates of school level growth
Empircal Bayes Estimates of School-Level Growth Vertical Scale

r = .96

  • Standard Approach in Colorado:

  • Separate

  • 3PLM/GPCM

  • EAP

Switch to Hybrid calibration


Empircal bayes estimates of school level growth1
Empircal Bayes Estimates of School-Level Growth Vertical Scale

r = .88

  • Standard Approach in Colorado:

  • Separate

  • 3PLM/GPCM

  • EAP

Switch to Hybrid calibration & 1PLM/GPCM


Empircal bayes estimates of school level growth2
Empircal Bayes Estimates of School-Level Growth Vertical Scale

r = .75

  • Standard Approach in Colorado:

  • Separate

  • 3PLM/GPCM

  • EAP

Switch to Hybrid calibration,1PLM/GPCM, MLE


Curtains down
Curtains Down Vertical Scale

  • Vertical scales have (largely) arbitrary metrics.

  • Absolute interpretations of growth can deceive.

    • Students might appear to grow “faster” solely because of the scaling approach.

    • Does criterion-referencing (i.e., standard-setting) take this into account?

  • A better approach might focus on changes in norm-referenced interpretations—but this conflicts with the NCLB perspective on growth.


ad