Download Presentation

A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options

A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options

594 Views

Download Presentation
## A Cognitive Diagnosis Model for Cognitively-Based Multiple-Choice Options

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**A Cognitive Diagnosis Model forCognitively-Based**Multiple-Choice Options Jimmy de la Torre Department of Educational Psychology Rutgers, The State University of New Jersey**All wrong answers are wrong;**But some wrong answers are more wrong than others.**Introduction**• Assessments should educate and improve student performance, not merely audit it • In other words, assessments should not only ascertain the status of learning, but also further learning • Due to emphasis on accountability, more and more resources are allocated towards assessments that only audit learning • Tests used to support school and system accountability do not provide diagnostic information about individual students**Tests based on unidimensional IRT models report**single-valued scores that submerge any distinct skills • These scores are useful in establishing relative order but not evaluation of students' specific strengths and weaknesses • Cluster scores have been used, but these scores are unreliable and provide superficial information about the underlying processes • Needed are assessments that can provide interpretative, diagnostic, highly informative, and potentially prescriptive information**Some psychometric models allow the merger of advances in**cognitive and psychometric theories to provide inferences more relevant to learning • These models are called cognitive diagnosis models (CDMs) • CDMs are discrete latent variable models • They are developed specifically for diagnosing the presence or absence of multiple fine-grained skills, processes or problem-solving strategies involved in an assessment**Fundamental difference between IRT and CDM: A fraction**subtraction example • IRT: performance is based on a unidimensional continuous latent trait • Students with higher latent traits have higher probability of answering the question correctly**Fundamental difference between IRT and CDM: A fraction**subtraction example • IRT: performance is based on a unidimensional continuous latent trait • Students with higher latent traits have higher probability of answering the question correctly • CDM: performance is based on binary attribute vector • Successful performance on the task requires a series of successful implementations of the attributes specified for the task**Required attributes:**(1) Borrowing from whole (2) Basic fraction subtraction (3) Reducing • Other attributes: (4) Separating whole from fraction (5) Converting whole to fraction**1**0.75 0.5 0.25 0**Background**• Denote the response and attribute vectors of examinee i by and • Each attribute pattern is a unique latent class; thus, K attributes define latent classes • Attribute specification for the items can be found in the Q-matrix, a J x K binary matrix • DINA (Deterministic Input Noisy “And” gate) is a CDM model that can be used in modeling the distribution of given**In the DINA model**• where is the latent group classification of examinee i with respect to item j • P(H|g) is the probability that examinees in group g will respond with h to item j • In more conventional notation of the DINA = guessing, = slip**Of the various test formats, multiple-choice (MC) has been**widely used for its ability to sample and accommodate diverse contents • Typical CDM analyses of MC tests involve dichotomized scores (i.e., correct/incorrect) • The approach ignores the diagnostic insights about student difficulties and alternative conceptions in the distractors • Wrong answers can reveal both what students know and what they do not know**Purpose of the paper is to propose a two-component framework**for maximizing the diagnostic value of MC assessments • Component 1: Prescribes how MC options can be designed to contain more diagnostic information • Component 2: Describes a CDM model that can exploit such information • Viability (i.e., estimability, efficiency) of the proposed framework is evaluated using a simulation study**Component 1: Cognitively-Based MC Options**• For the MC format, , where each number represents a different option • An option is coded or cognitively-based if it is constructed to correspond to some of the latent classes • Each coded option has an attribute specification • Attribute specifications for non-coded options are implicitly represented by the zero-vector**A Fraction Subtraction Example**A) B) C) D)**The option with the largest number of required attributes is**the key**The option with the largest number of required attributes is**the key • Distractors are created to reflect the type of responses students who lack one or more of the required attributes for the key are likely to give**The option with the largest number of required attributes is**the key • Distractors are created to reflect the type of responses students who lack one or more of the required attributes for the key are likely to give • Knowledge states represented by the distractors should be in the subset of the knowledge state that corresponds to the key • Number of latent classes under the proposed framework is equal to , the number of coded options plus 1**000**100 010 001 110 101 011 111 “0”**“0”**“1” 000 100 010 001 110 101 011 111**“1”**“2” “3” 000 100 010 001 110 101 011 111**“1”**“2” “4” “3” “0” 000 100 010 001 110 101 011 111**Component 2: The MC-DINA Model**• Let be the Q-vector for option h of item j, and • With respect to item j, examinee i is in group • Probability of examinee i choosing option h of item j is**0**1 2 3 4**DINA Model for Nominal Response**N-DINA Model**Group**A C D B 0 1**P(1|0) – guessing parameter**P(0|1) – slip parameter Plain DINA Model**A Simulation Study**• Purpose: To investigate how • well the item parameters and SE can be estimated • accurately the attributes can be classified • MC-DINA compares with the traditional DINA • 1000 examinees, 30 items, 5 attributes • Parameters: • Number of replicates: 100**Results**Bias, Mean and Empirical SE Across 30 Items**Attribute Classification Accuracy**Percent of Attribute Correctly Classified 89.71 97.43 91.13 69.58 6.30 20.13**Summary and Conclusion**• There is an urgent need for assessments that provide interpretative, diagnostic, highly informative, and potentially prescriptive scores • This type of scores can inform classroom instruction and learning • With appropriate construction, MC items can be designed to be more diagnostically informative • Diagnostic information in MC distractors can be harnessed using the MC-DINA**Parameters of the MC-DINA model can be accurately estimated**• MC-DINA attribute classification accuracy is dramatically better than the traditional DINA • Caveat: This framework is only the psychometric aspect of cognitive diagnosis • Development of cognitively diagnostic assessment is a multi-disciplinary endeavor requiring collaboration between experts from learning science, cognitive science, subject domains, didactics, psychometrics, . . .