The Role of Cognition in Educational Assessment Design - PowerPoint PPT Presentation

The role of cognition in educational assessment design
1 / 43

  • Uploaded on
  • Presentation posted in: General

The Role of Cognition in Educational Assessment Design. Joanna S. Gorin Arizona State University. Overview. The need for cognitively-based assessment. Defining cognitive models and their properties. Tools for cognitively-based assessment design and analysis.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

The Role of Cognition in Educational Assessment Design

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

The role of cognition in educational assessment design

The Role of Cognition in Educational Assessment Design

Joanna S. Gorin

Arizona State University



  • The need for cognitively-based assessment.

  • Defining cognitive models and their properties.

  • Tools for cognitively-based assessment design and analysis.

  • Empirical research on cognitive approaches to test design and validation.

The purpose of educational assessment

The Purpose of Educational Assessment

  • New purposes for testing have introduced issues related to the inappropriateness of current standardized tests.

    Such assessments shall produce individual student interpretive, descriptive, and diagnostic reports…that allow parents, teachers, and principals to understand and address the specific academic needs of students, and include information regarding achievement on academic assessments aligned with State academic achievement standards, and that are provided to parents, teachers, and principals as soon as is practicably possible after the assessment is given, in an understandable and uniform format, and to the extent practicable, in a language that parents can understand.

    NCLB Part A Subpart 1 Sec. 2221(b)3(C)(xii), 2001

Implications of new assessment needs

Implications of new assessment needs.

  • “[a]ll assessments will be more fruitful when based on an understanding of cognition in the domain and on the precept of reasoning with evidence” (NRC, 2001, p. 178)

    • Increase our understanding of the claims that we want to make about students, instruction, programs, and policy.

      • Detailed description of the skills, abilities, constructs we are measuring.

    • Increased understanding of the evidence provided by the student responses to the test questions.

      • Detailed description of item difficulty, discrimination, and other statistical and psychological properties.

Cognitively based assessment design

Cognitively Based Assessment Design

  • If we begin with a complete understanding of the skill, domain, competency, ability, etc. that we want to measure, then we can be more principled in our design of the tools we use to measure it.

  • Further, if we understand our measurement tools and the scores they generate more fully, we can evaluate their quality relative to our measurement goals.

Cognitive models and grain size

Cognitive Models and Grain Size

  • What is a complete understanding of the construct? Of the test?

  • Types of models:

    • Model of test specifications (content standards).

    • Model of task performance (cognitive processes).

  • Alignment of the test with standards or test specifications may not be sufficient to yield the desired information.

    • The model and alignment must be made at the appropriate level of inference.

General model of cognitively based assessment design

Construct / Latent Trait

Process A

Process B

Process C

Process D

Feature A

Feature B

Feature C

Feature D

Item / Test Question

General Model of Cognitively-Based Assessment Design

Potential cognitive models

Potential Cognitive Models

  • Information processing models.

    • Coherence-integration theory of text processing.

      • Propositional representation of text.

      • Cyclic construction of representation and integration of new information.

    • Activation theory.

      • Activation of information is influenced by various factors.

      • The most highly activated information will be selected.

  • Evaluate the “completeness” of these models in terms of the target inferences to be made about students.

Steps in assessment design

Steps in Assessment Design

Construct-based Model

Tools for cognitively based assessment development

Tools for Cognitively-Based Assessment Development

  • Design Frameworks

    • Mislevy’s Evidence Centered Design

      • A multi-level model of assessment.

    • Embretson’s Cognitive Design System

      • A process-based approach.

    • Bennett and Bejar’s Generative Approach

      • A structural approach.

Cognitively based assessment design1

Cognitively-Based Assessment Design

Task-based Model

Purpose of item difficulty modeling idm

Purpose of Item Difficulty Modeling (IDM)

  • Extend available student and item information beyond a single statistical parameter.

  • Substantive information can be useful for:

    • Verification of construct definition (Construct Validity).

    • Creating new items (Automatic/Algorithmic Item Generation).

    • Providing diagnostic information (Score Reporting).

    • Understanding group differences (DIF).

Tools for cognitively based assessment development1

Tools for Cognitively-Based Assessment Development

  • Design Frameworks for Test Construction

    • Evidence Centered Design

    • Cognitive Design System

    • Generative Approach

  • Psychometric Models for IDM

    • Tree Based Regression Approach

    • Rule Space Methodology

    • Attribute Hierarchy Method

    • Fusion Model

    • Linear Logistic Latent Trait Model

  • Traditional item construct definition

    Traditional item-construct definition.

    • Reading comprehension questions measure the ability to read with understanding, insight and discrimination. This type of question explores the ability to analyze a written passage from several perspectives. These include the ability to recognize both explicitly stated elements in the passage and assumptions underlying statements or arguments in the passage as well as the implications of those statements or arguments.

    Model of test specifications standards and objectives

    Model of Test-Specifications (Standards and Objectives)

    Cognitive model development

    Cognitive Model Development

    • Cognitive Theory

      • Generate list of relevant processing components from theory.

  • Correlational Studies

    • Establish a statistical relationship between the features and the item properties.

  • Experimental Manipulations

    • Context/format

    • Item Design

  • Use process tracing methods to identify additional processing influences.

    • Verbal protocols (“think alouds”)

    • Eye-tracking data

  • Cognitive model skill subskill model finer grain size

    Cognitive Model (Skill/Subskill Model)finer grain size

    Cognitive model information processing model

    Cognitive Model (Information Processing Model)

    Cognitive variables

    Cognitive Variables

    • Modifier Propositional Density

    • Predicate Propositional Density

    • Text Content Vocabulary Level

    • Percent Content Words

    • Percent of Relevant Text

    • Falsification

    • Confirmation

    • Vocabulary Level of the Distractors

    • Vocabulary Level of the Correct Response

    • Reasoning of the Distractors

    • Reasoning of the Correct Response

    • Location of Relevant Information in Text

    • Length of Passage

    • Special Item Format

    Full item difficulty model

    • Vocabulary Level

    • Sentence Length

    • Propositional Density

    • Argument Structure

    • Text Length

    • Vocabulary Level

    • Sentence Length

    • Semantic Overlap

    • Level of Question

    • Vocabulary Level of Key and Distractors

    • Falsifiability of Distractors

    • Confirmation of the Key

    Full Item Difficulty Model

    Activation model of item difficulty

    Activation Model of Item Difficulty

    Regression model of gre items

    Regression Model of GRE Items

    Contribution of processing factors to item difficulty

    Contribution of Processing Factors to Item Difficulty

    Implications for score meaning

    Implications for Score Meaning

    • Variables related to both text encoding and decision processes were significant predictors in models of GRE-V item difficulty.

    • This suggests that GRE-V RC items measure both processes.

      • New evidence on construct validity of test scores for these items.

    Experimental manipulations

    Experimental Manipulations

    • Experimental conditions corresponded to variations in item features based on a hypothesized cognitive model.

      • Passage propositional density and syntax modification.

      • Passage passive voice and negative wording modification.

      • Passage order of information change.

      • Response alternative-passage overlap change.

    • Experimental effects tested with the LLTM model.

      • Rasch model to deconstruct sources of item difficulty.

    Contrast coding for lltm analysis

    Contrast Coding for LLTM Analysis

    Scatterplot of known and estimated item difficulty parameters for 29 original

    Scatterplot of Known and Estimated Item Difficulty Parameters for 29-Original

    Effects of manipulations on item difficulty

    Effects of Manipulations on Item Difficulty

    Table of regression coefficients and significance tests for the experimental model

    Table of Regression Coefficients and Significance Tests for the Experimental Model

    Implications for test design

    Implications for Test Design

    • Significant effect of passive voice and negative wording on item difficulty was found.

      • Test writers often avoid the use of negative wording, citing that it is complicated and can confuse readers.

    • Although the effects were not significant for item difficulty, two significant effects on response time were found.

    Self report measures

    Self Report Measures

    • Verbal Protocols

      • Concurrent or retrospective

    • Structured Questionnaires

      • Strategy use

      • Background information

      • Interest

      • Confidence

    Digital eye tracking

    Digital Eye Tracking

    • Digital eye tracking data has been used to examine cognition and individual differences in

      • language processing.

      • facial processing.

      • learned attention.

      • electrical circuit troubleshooting.

      • problem solving strategies.

        • Spatial reasoning

        • Abstract reasoning

    The role of cognition in educational assessment design

    Look Zone 1

    Look Zone 2

    Look Zone 3

    The role of cognition in educational assessment design

    Look Zone 1

    Look Zone 2

    Look Zone 3

    Gazetrail for a reading comprehension item

    Gazetrail for a Reading Comprehension Item

    • Question (0.797)

    • RO (0.297)

    • Passage (4.03)

    • Question (0.31)

    • Passage (41.35)

    • Question (2.59)

    • Passage (2.65)

    • RO (6.00)

    • Passage (3.09)

    • RO (0.76)

    • Question (0.37)

    • RO (.25)

    Summary data for reading comprehension item

    Summary Data for Reading Comprehension Item

    Individual vs group differences in item processing

    Individual vs Group Differences in Item Processing

    Point of gaze by time

    Point of Gaze by Time

    Implications and future use

    Implications and future use.

    • Verify some of our current models.

    • Identify new variables related to processing.

    • Describe qualitative differences characterized by strategy differences.

    • Examine specific aspects of test items that are problematic for individuals or for subgroups.

    • Observe the effects of controlled item manipulations on item processing, not just item responses alone.

    Summary of the benefits of cognitively based assessment design

    Summary of the Benefits of Cognitively-Based Assessment Design

    • Construct validity is more completely understood.

      • Explicitly elaborates the processes, strategies and knowledge structures.

    • Enhanced score interpretations.

      • Persons, as well as items, can be described by processes, strategies and knowledge structures.

    • Generation of items with specified sources and levels of item difficulty.

      • Item parameters may be predicted for newly developed items.

      • Items can be generated for specific populations by controlling the cognitive processing requirement.

    Greatest challenges

    Greatest Challenges

    • Our limited understanding of the cognitive models and the test items.

      • Current item response data is limited in information.

    • A “one size fits all” approach to cognitive modeling will not work.

      • Changing your goals for the assessment necessitates a change in the items.

      • Changing the items (or features of the items) implies a change in what can be concluded from the test.

  • Login