a cognitive diagnosis model for cognitively based multiple choice options l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options PowerPoint Presentation
Download Presentation
A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options

Loading in 2 Seconds...

play fullscreen
1 / 46

A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options - PowerPoint PPT Presentation


  • 567 Views
  • Uploaded on

A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options Jimmy de la Torre Department of Educational Psychology Rutgers, The State University of New Jersey All wrong answers are wrong; But some wrong answers are more wrong than others. Introduction

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options' - Audrey


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a cognitive diagnosis model for cognitively based multiple choice options
A Cognitive Diagnosis Model forCognitively-Based Multiple-Choice Options

Jimmy de la Torre

Department of Educational Psychology

Rutgers, The State University of New Jersey

slide2

All wrong answers are wrong;

But some wrong answers

are more wrong than others.

introduction
Introduction
  • Assessments should educate and improve student performance, not merely audit it
  • In other words, assessments should not only ascertain the status of learning, but also further learning
  • Due to emphasis on accountability, more and more resources are allocated towards assessments that only audit learning
  • Tests used to support school and system accountability do not provide diagnostic information about individual students
slide4
Tests based on unidimensional IRT models report single-valued scores that submerge any distinct skills
  • These scores are useful in establishing relative order but not evaluation of students' specific strengths and weaknesses
  • Cluster scores have been used, but these scores are unreliable and provide superficial information about the underlying processes
  • Needed are assessments that can provide interpretative, diagnostic, highly informative, and potentially prescriptive information
slide5
Some psychometric models allow the merger of advances in cognitive and psychometric theories to provide inferences more relevant to learning
  • These models are called cognitive diagnosis models (CDMs)
  • CDMs are discrete latent variable models
  • They are developed specifically for diagnosing the presence or absence of multiple fine-grained skills, processes or problem-solving strategies involved in an assessment
slide6
Fundamental difference between IRT and CDM: A fraction subtraction example
  • IRT: performance is based on a unidimensional continuous latent trait
  • Students with higher latent traits have higher probability of answering the question correctly
slide8
Fundamental difference between IRT and CDM: A fraction subtraction example
  • IRT: performance is based on a unidimensional continuous latent trait
  • Students with higher latent traits have higher probability of answering the question correctly
  • CDM: performance is based on binary attribute vector
  • Successful performance on the task requires a series of successful implementations of the attributes specified for the task
slide9

Required attributes:

(1) Borrowing from whole

(2) Basic fraction subtraction

(3) Reducing

  • Other attributes:

(4) Separating whole from fraction

(5) Converting whole to fraction

slide10

1

0.75

0.5

0.25

0

slide11

Background

  • Denote the response and attribute vectors of examinee i by and
  • Each attribute pattern is a unique latent class; thus, K attributes define latent classes
  • Attribute specification for the items can be found in the Q-matrix, a J x K binary matrix
  • DINA (Deterministic Input Noisy “And” gate) is a CDM model that can be used in modeling the distribution of given
slide12
In the DINA model
  • where

is the latent group classification of examinee i with respect to item j

  • P(H|g) is the probability that examinees in group g will respond with h to item j
  • In more conventional notation of the DINA

= guessing, = slip

slide13
Of the various test formats, multiple-choice (MC) has been widely used for its ability to sample and accommodate diverse contents
  • Typical CDM analyses of MC tests involve dichotomized scores (i.e., correct/incorrect)
  • The approach ignores the diagnostic insights about student difficulties and alternative conceptions in the distractors
  • Wrong answers can reveal both what students know and what they do not know
slide14
Purpose of the paper is to propose a two-component framework for maximizing the diagnostic value of MC assessments
  • Component 1: Prescribes how MC options can be designed to contain more diagnostic information
  • Component 2: Describes a CDM model that can exploit such information
  • Viability (i.e., estimability, efficiency) of the proposed framework is evaluated using a simulation study
component 1 cognitively based mc options
Component 1: Cognitively-Based MC Options
  • For the MC format, , where each number represents a different option
  • An option is coded or cognitively-based if it is constructed to correspond to some of the latent classes
  • Each coded option has an attribute specification
  • Attribute specifications for non-coded options are implicitly represented by the zero-vector
slide20
The option with the largest number of required attributes is the key
  • Distractors are created to reflect the type of responses students who lack one or more of the required attributes for the key are likely to give
slide22
The option with the largest number of required attributes is the key
  • Distractors are created to reflect the type of responses students who lack one or more of the required attributes for the key are likely to give
  • Knowledge states represented by the distractors should be in the subset of the knowledge state that corresponds to the key
  • Number of latent classes under the proposed framework is equal to , the number of coded options plus 1
slide23

000

100

010

001

110

101

011

111

“0”

slide24

“0”

“1”

000

100

010

001

110

101

011

111

slide25

“1”

“2”

“3”

000

100

010

001

110

101

011

111

slide26

“1”

“2”

“4”

“3”

“0”

000

100

010

001

110

101

011

111

component 2 the mc dina model
Component 2: The MC-DINA Model
  • Let be the Q-vector for option h of item j, and
  • With respect to item j, examinee i is in group
  • Probability of examinee i choosing option h of item j is
slide35

0

1

2

3

4

slide38

Group

A

C

D

B

0

1

slide40

P(1|0) – guessing parameter

P(0|1) – slip parameter

Plain DINA Model

a simulation study
A Simulation Study
  • Purpose: To investigate how
    • well the item parameters and SE can be estimated
    • accurately the attributes can be classified
    • MC-DINA compares with the traditional DINA
  • 1000 examinees, 30 items, 5 attributes
  • Parameters:
  • Number of replicates: 100
results
Results

Bias, Mean and Empirical SE Across 30 Items

attribute classification accuracy
Attribute Classification Accuracy

Percent of Attribute Correctly Classified

89.71

97.43

91.13

69.58

6.30

20.13

summary and conclusion
Summary and Conclusion
  • There is an urgent need for assessments that provide interpretative, diagnostic, highly informative, and potentially prescriptive scores
  • This type of scores can inform classroom instruction and learning
  • With appropriate construction, MC items can be designed to be more diagnostically informative
  • Diagnostic information in MC distractors can be harnessed using the MC-DINA
slide45
Parameters of the MC-DINA model can be accurately estimated
  • MC-DINA attribute classification accuracy is dramatically better than the traditional DINA
  • Caveat: This framework is only the psychometric aspect of cognitive diagnosis
  • Development of cognitively diagnostic assessment is a multi-disciplinary endeavor requiring collaboration between experts from learning science, cognitive science, subject domains, didactics, psychometrics, . . .