1 / 29

Introduction to plausible values

Introduction to plausible values. ICCS IDB Training Seminar Hamburg, 24-26 November 2010. Content of presentation. Rationale for scaling Rasch model and possible ability estimates Shortcomings of point estimates Drawing plausible values Computation of measurement error.

Download Presentation

Introduction to plausible values

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Introduction to plausible values ICCS IDB Training Seminar Hamburg, 24-26 November 2010

  2. Content of presentation • Rationale for scaling • Rasch model and possible ability estimates • Shortcomings of point estimates • Drawing plausible values • Computation of measurement error

  3. Rationale for IRT scaling of data • Summarising data instead of dealing with many single items • Raw scores or percent correct sample-dependent • Makes equating possible and can deal with rotated test forms

  4. The ‘Rasch model’ • Models the probability to respond correctly to an item as • Likewise, the probability of NOT responding correctly is modelled as

  5. IRT curves

  6. How might we impute a reasonable proficiency value? • Choose the proficiency that makes the score most likely • Maximum Likelihood Estimate • Weighted Likelihood Estimate • Choose the most likely proficiency for the score • empirical Bayes • Choose a selection of likely proficiencies for the score • Multiple imputations (plausible values)

  7. Maximum Likelihood vs. Raw Score

  8. Score 3 Score 4 Score 5 Score 6 Score 2 Score 1 Score 0 Proficiency on Logit Scale The Resulting Proficiency Distribution

  9. Characteristics of Maximum Likelihood Estimates (MLE) • Unbiased at individual level with sufficient information BUT biased towards ends of ability scale. • Arbitrary treatment of perfects and zeroes required • Discrete scale & measurement error leads to bias in population parameter estimates

  10. Characteristics of Weighted Likelihood Estimates • Less biased than MLE • Provides estimates for perfect and zero scores • BUT discrete scale & measurement error leads to bias in population parameter estimates

  11. Plausible Values • What are plausible values? • Why do we use them? • How to analyse plausible values?

  12. Purpose of educational tests • Measure particular students(minimise measurement error of individual estimates) • Assess populations(minimise error when generalising to the population)

  13. Posterior distributionsfor test scores on 6 dichotomous items

  14. Empirical Bayes – Expected A-Priori estimates (EAP)

  15. Characteristics of EAPs • Biased at the individual level but unbiased population means (NOT variances) • Discrete scale, bias & measurement error leads to bias in population parameter estimates • Requires assumptions about the distribution of proficiency in the population

  16. Score 4 Score 5 Score 6 Score 0 Score 1 Score 2 Score 3 Proficiency on Logit Scale Plausible Values

  17. Characteristics of Plausible Values • Not fair at the student level • Produces unbiased population parameter estimates • if assumptions of scaling are reasonable • Requires assumptions about the distribution of proficiency

  18. Estimating percentages below benchmark with Plausible Values Level One Cutpoint The proportion of plausible values less than the cut-point will be a superior estimator to the EAP, MLE or WLE based values

  19. Methodology of PVs • Mathematically computing posterior distributions around test scores • Drawing 5 random values for each assessed individual from the posterior distribution for that individual

  20. What is conditioning? • Assuming normal posterior distribution: • Model sub-populations:X=0 for boyX=1 for girl

  21. Conditioning Variables • Plausible values should only be analysed with data that were included in the conditioning (otherwise, results may be biased) • Aim: Maximise information included in the conditioning, that is use as many variables as possible • To reduce number of conditioning variables, factor scores from principal component analysis were used in ICCS • Use of classroom dummies takes between-school variation into account (no inclusion of school or teacher questionnaire data needed)

  22. Plausible values • Model with conditioning variables will improve precision of prediction of ability (population estimates ONLY) • Conditioning provides unbiased estimates for modelled parameters. • Simulation studies comparing PVs, EAPs and WLEs show that • Population means similar results • WLEs (or MLEs) tend to overestimate variances • EAPs tend to underestimate variance

  23. Calculating of measurement error • As in TIMSS or PIRLS data files, there are five plausible values for cognitive test scales in ICCS • Using five plausible values enable researchers to obtain estimates of the measurement error

  24. How to analyse PVs - 1 • Estimated mean is the AVERAGE of the mean for each PV • Sampling variance is the AVERAGE of the sampling variance for each PV

  25. How to analyse PVs - 2 • Measurement variance computed as: • Total standard error computed from measurement and sampling variance as:

  26. How to analyse PVs - 3 • can be replaced by any statistic for instance:- SD- Percentile- Correlation coefficient- Regression coefficient- R-square- etc.

  27. Steps for estimating both sampling and measurement error • Compute statistic for each PV for fully weighted sample • Compute statistics for each PV for 75 replicate samples • Compute sampling error (based on previous steps) • Compute measurement error • Combine error variances to calculate standard error

  28. Software that can handle PVs • Tailored macros (e.g. in SAS/SPSS) • IDB Analyser • SPSS Replicates module (ACER) • WESVAR • AM • HLM (for multilevel modelling)

  29. Questions or comments?

More Related