Should we report subscores
Download
1 / 20

Should we report subscores? - PowerPoint PPT Presentation


  • 69 Views
  • Uploaded on

Should we report subscores?. Subscore as Skills Diagnostic information Cognitive Diagnostic Models (CDMs) Dimensionality Analysis Classical Test Theory based subscore analysis Analysis of Classification Accuracy based on synthesized empirical data. Outline. Why use subscore?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Should we report subscores?' - meadow


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Outline

Subscore as Skills Diagnostic information

Cognitive Diagnostic Models (CDMs)

Dimensionality Analysis

Classical Test Theory based subscore analysis

Analysis of Classification Accuracy based on synthesized empirical data

Outline


Subscores as skills diagnostic information

Why use subscore?

Can provide more useful information about teaching and learning

Example:

Rather than reporting Candy being in the 80th percentile in statistics…

Report Candy is proficient in multivariate analysis, experimental design, but needs to work on time series, sampling.

Follow up such profile with quality individualized instruction for Candy.

Subscores as Skills Diagnostic information


Cognitive diagnostic models

Fusion model

Simplified from Unified model (DiBello et al., 1995)

Good side

Can link the observable performance to latent skills specified by expert/theory/previous experience (Q-matrix)

Limitation

Estimation

Parameters can be estimated using Hierarchical Bayesian approach (Arpeggio).

Cognitive Diagnostic Models


Dimensionality analysis

To find the dimension structure of the test

DETECT and NOHARM

Dimensionality Analysis


Classical test theory based subscore analysis

Goal: To predict the “true subscore” from the observed score

Predictors of true subscore

Observed subscore

Observed total score

To report subscore in a test?

Classical Test Theory based subscore analysis


Method classical test theory
Method: Classical Test Theory score

  • Haberman’s Method (Haberman, 2008)

  • Proportional Reduction in MSE (PRMSE)


Research questions

Q1: score

Can raw subscore obtain comparative student skill profiles as fusion model does?

Q2:

Can four subscores (observed subscore, mastery probability, and expected subscores based on CTT, MLE of θ) produce comparative classification accuracy?

Research Questions


Analysis procedure for q1

Step 1: Dimensionality analysis score

Step 2: Cluster analysis

Derive profiles based on raw subscores, transformed raw subscores, and mastery probability

Step 3: Comparison of profiles by cluster analysis

Analysis procedure for Q1


Sample size: 14874 score

Number of subtests: 3

Content area : Mathematics, Reading, Science

Data


Results of dimensionality analysis
Results of Dimensionality Analysis score

  • DETECT

    DETECT index: 0.22

  • NOHARM

    Correlation between dimensions


Results of cluster analysis not in same scale
Results of Cluster analysis (NOT in same scale) score

  • Comparison of cluster solutions


Results of cluster analysis in same scale
Results of Cluster analysis ( in same scale) score

  • Comparison of cluster solutions


Analysis procedure for q2

Step 1: Set the cutting scores based on scoreθcuts

Step 2: Compute the subscores from each method

Observed subscore (obs)

Expected subscore based on either observed subscore or total score (hab)

Maximum-likelihood estimation of θ(mle)

Posterior probability of mastery (fus)

Step 3: Classify students to be proficiency or not based on the cutting scores

Step 4: Calculate the exact agreement among the classifications of each method

Analysis procedure for Q2


Haberman 2007 analysis based upon classical test theory
Haberman (2007) analysis based upon scoreClassical Test Theory

  • Summary statistics for Subscores


Analysis of mastery non mastery classification
Analysis of mastery/non-mastery classification score

  • Classification for Math


Analysis of mastery non mastery classification1
Analysis of mastery/non-mastery classification score

  • Classification for Reading


Analysis of mastery non mastery classification2
Analysis of mastery/non-mastery classification score

  • Classification for Science



Future study

Simulation study 1: How does dimension structure affect the similarity of profiles based on raw subscore and mastery probability?

Study 2: Can the standard setting information enhance the classification accuracy of fusion model?

Future Study


ad