1 / 20

Should we report subscores?

Should we report subscores?. Subscore as Skills Diagnostic information Cognitive Diagnostic Models (CDMs) Dimensionality Analysis Classical Test Theory based subscore analysis Analysis of Classification Accuracy based on synthesized empirical data. Outline. Why use subscore?

meadow
Download Presentation

Should we report subscores?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Should we report subscores?

  2. Subscore as Skills Diagnostic information Cognitive Diagnostic Models (CDMs) Dimensionality Analysis Classical Test Theory based subscore analysis Analysis of Classification Accuracy based on synthesized empirical data Outline

  3. Why use subscore? Can provide more useful information about teaching and learning Example: Rather than reporting Candy being in the 80th percentile in statistics… Report Candy is proficient in multivariate analysis, experimental design, but needs to work on time series, sampling. Follow up such profile with quality individualized instruction for Candy. Subscores as Skills Diagnostic information

  4. Fusion model Simplified from Unified model (DiBello et al., 1995) Good side Can link the observable performance to latent skills specified by expert/theory/previous experience (Q-matrix) Limitation Estimation Parameters can be estimated using Hierarchical Bayesian approach (Arpeggio). Cognitive Diagnostic Models

  5. To find the dimension structure of the test DETECT and NOHARM Dimensionality Analysis

  6. Goal: To predict the “true subscore” from the observed score Predictors of true subscore Observed subscore Observed total score To report subscore in a test? Classical Test Theory based subscore analysis

  7. Method: Classical Test Theory • Haberman’s Method (Haberman, 2008) • Proportional Reduction in MSE (PRMSE)

  8. Q1: Can raw subscore obtain comparative student skill profiles as fusion model does? Q2: Can four subscores (observed subscore, mastery probability, and expected subscores based on CTT, MLE of θ) produce comparative classification accuracy? Research Questions

  9. Step 1: Dimensionality analysis Step 2: Cluster analysis Derive profiles based on raw subscores, transformed raw subscores, and mastery probability Step 3: Comparison of profiles by cluster analysis Analysis procedure for Q1

  10. Sample size: 14874 Number of subtests: 3 Content area : Mathematics, Reading, Science Data

  11. Results of Dimensionality Analysis • DETECT DETECT index: 0.22 • NOHARM Correlation between dimensions

  12. Results of Cluster analysis (NOT in same scale) • Comparison of cluster solutions

  13. Results of Cluster analysis ( in same scale) • Comparison of cluster solutions

  14. Step 1: Set the cutting scores based on θcuts Step 2: Compute the subscores from each method Observed subscore (obs) Expected subscore based on either observed subscore or total score (hab) Maximum-likelihood estimation of θ(mle) Posterior probability of mastery (fus) Step 3: Classify students to be proficiency or not based on the cutting scores Step 4: Calculate the exact agreement among the classifications of each method Analysis procedure for Q2

  15. Haberman (2007) analysis based upon Classical Test Theory • Summary statistics for Subscores

  16. Analysis of mastery/non-mastery classification • Classification for Math

  17. Analysis of mastery/non-mastery classification • Classification for Reading

  18. Analysis of mastery/non-mastery classification • Classification for Science

  19. Summary of Classification analysis

  20. Simulation study 1: How does dimension structure affect the similarity of profiles based on raw subscore and mastery probability? Study 2: Can the standard setting information enhance the classification accuracy of fusion model? Future Study

More Related