Should we report subscores?. Subscore as Skills Diagnostic information Cognitive Diagnostic Models (CDMs) Dimensionality Analysis Classical Test Theory based subscore analysis Analysis of Classification Accuracy based on synthesized empirical data. Outline. Why use subscore?
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Cognitive Diagnostic Models (CDMs)
Classical Test Theory based subscore analysis
Analysis of Classification Accuracy based on synthesized empirical dataOutline
Can provide more useful information about teaching and learning
Rather than reporting Candy being in the 80th percentile in statistics…
Report Candy is proficient in multivariate analysis, experimental design, but needs to work on time series, sampling.
Follow up such profile with quality individualized instruction for Candy.Subscores as Skills Diagnostic information
Simplified from Unified model (DiBello et al., 1995)
Can link the observable performance to latent skills specified by expert/theory/previous experience (Q-matrix)
Parameters can be estimated using Hierarchical Bayesian approach (Arpeggio).Cognitive Diagnostic Models
DETECT and NOHARMDimensionality Analysis
Predictors of true subscore
Observed total score
To report subscore in a test?Classical Test Theory based subscore analysis
Can raw subscore obtain comparative student skill profiles as fusion model does?
Can four subscores (observed subscore, mastery probability, and expected subscores based on CTT, MLE of θ) produce comparative classification accuracy?Research Questions
Step 2: Cluster analysis
Derive profiles based on raw subscores, transformed raw subscores, and mastery probability
Step 3: Comparison of profiles by cluster analysisAnalysis procedure for Q1
Sample size: 14874 score
Number of subtests: 3
Content area : Mathematics, Reading, ScienceData
DETECT index: 0.22
Correlation between dimensions
Step 1: Set the cutting scores based on scoreθcuts
Step 2: Compute the subscores from each method
Observed subscore (obs)
Expected subscore based on either observed subscore or total score (hab)
Maximum-likelihood estimation of θ(mle)
Posterior probability of mastery (fus)
Step 3: Classify students to be proficiency or not based on the cutting scores
Step 4: Calculate the exact agreement among the classifications of each methodAnalysis procedure for Q2
Simulation study 1: How does dimension structure affect the similarity of profiles based on raw subscore and mastery probability?
Study 2: Can the standard setting information enhance the classification accuracy of fusion model?Future Study