The role of cognition in educational assessment design
1 / 43

The Role of Cognition in Educational Assessment Design - PowerPoint PPT Presentation

  • Uploaded on

The Role of Cognition in Educational Assessment Design. Joanna S. Gorin Arizona State University. Overview. The need for cognitively-based assessment. Defining cognitive models and their properties. Tools for cognitively-based assessment design and analysis.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'The Role of Cognition in Educational Assessment Design' - kamil

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The role of cognition in educational assessment design

The Role of Cognition in Educational Assessment Design

Joanna S. Gorin

Arizona State University


  • The need for cognitively-based assessment.

  • Defining cognitive models and their properties.

  • Tools for cognitively-based assessment design and analysis.

  • Empirical research on cognitive approaches to test design and validation.

The purpose of educational assessment
The Purpose of Educational Assessment

  • New purposes for testing have introduced issues related to the inappropriateness of current standardized tests.

    Such assessments shall produce individual student interpretive, descriptive, and diagnostic reports…that allow parents, teachers, and principals to understand and address the specific academic needs of students, and include information regarding achievement on academic assessments aligned with State academic achievement standards, and that are provided to parents, teachers, and principals as soon as is practicably possible after the assessment is given, in an understandable and uniform format, and to the extent practicable, in a language that parents can understand.

    NCLB Part A Subpart 1 Sec. 2221(b)3(C)(xii), 2001

Implications of new assessment needs
Implications of new assessment needs.

  • “[a]ll assessments will be more fruitful when based on an understanding of cognition in the domain and on the precept of reasoning with evidence” (NRC, 2001, p. 178)

    • Increase our understanding of the claims that we want to make about students, instruction, programs, and policy.

      • Detailed description of the skills, abilities, constructs we are measuring.

    • Increased understanding of the evidence provided by the student responses to the test questions.

      • Detailed description of item difficulty, discrimination, and other statistical and psychological properties.

Cognitively based assessment design
Cognitively Based Assessment Design

  • If we begin with a complete understanding of the skill, domain, competency, ability, etc. that we want to measure, then we can be more principled in our design of the tools we use to measure it.

  • Further, if we understand our measurement tools and the scores they generate more fully, we can evaluate their quality relative to our measurement goals.

Cognitive models and grain size
Cognitive Models and Grain Size

  • What is a complete understanding of the construct? Of the test?

  • Types of models:

    • Model of test specifications (content standards).

    • Model of task performance (cognitive processes).

  • Alignment of the test with standards or test specifications may not be sufficient to yield the desired information.

    • The model and alignment must be made at the appropriate level of inference.

General model of cognitively based assessment design

Construct / Latent Trait

Process A

Process B

Process C

Process D

Feature A

Feature B

Feature C

Feature D

Item / Test Question

General Model of Cognitively-Based Assessment Design

Potential cognitive models
Potential Cognitive Models

  • Information processing models.

    • Coherence-integration theory of text processing.

      • Propositional representation of text.

      • Cyclic construction of representation and integration of new information.

    • Activation theory.

      • Activation of information is influenced by various factors.

      • The most highly activated information will be selected.

  • Evaluate the “completeness” of these models in terms of the target inferences to be made about students.

Steps in assessment design
Steps in Assessment Design

Construct-based Model

Tools for cognitively based assessment development
Tools for Cognitively-Based Assessment Development

  • Design Frameworks

    • Mislevy’s Evidence Centered Design

      • A multi-level model of assessment.

    • Embretson’s Cognitive Design System

      • A process-based approach.

    • Bennett and Bejar’s Generative Approach

      • A structural approach.

Purpose of item difficulty modeling idm
Purpose of Item Difficulty Modeling (IDM)

  • Extend available student and item information beyond a single statistical parameter.

  • Substantive information can be useful for:

    • Verification of construct definition (Construct Validity).

    • Creating new items (Automatic/Algorithmic Item Generation).

    • Providing diagnostic information (Score Reporting).

    • Understanding group differences (DIF).

Tools for cognitively based assessment development1
Tools for Cognitively-Based Assessment Development

  • Design Frameworks for Test Construction

    • Evidence Centered Design

    • Cognitive Design System

    • Generative Approach

  • Psychometric Models for IDM

    • Tree Based Regression Approach

    • Rule Space Methodology

    • Attribute Hierarchy Method

    • Fusion Model

    • Linear Logistic Latent Trait Model

  • Traditional item construct definition
    Traditional item-construct definition.

    • Reading comprehension questions measure the ability to read with understanding, insight and discrimination. This type of question explores the ability to analyze a written passage from several perspectives. These include the ability to recognize both explicitly stated elements in the passage and assumptions underlying statements or arguments in the passage as well as the implications of those statements or arguments.

    Cognitive model development
    Cognitive Model Development

    • Cognitive Theory

      • Generate list of relevant processing components from theory.

  • Correlational Studies

    • Establish a statistical relationship between the features and the item properties.

  • Experimental Manipulations

    • Context/format

    • Item Design

  • Use process tracing methods to identify additional processing influences.

    • Verbal protocols (“think alouds”)

    • Eye-tracking data

  • Cognitive model skill subskill model finer grain size
    Cognitive Model (Skill/Subskill Model)finer grain size

    Cognitive model information processing model
    Cognitive Model (Information Processing Model)

    Cognitive variables
    Cognitive Variables

    • Modifier Propositional Density

    • Predicate Propositional Density

    • Text Content Vocabulary Level

    • Percent Content Words

    • Percent of Relevant Text

    • Falsification

    • Confirmation

    • Vocabulary Level of the Distractors

    • Vocabulary Level of the Correct Response

    • Reasoning of the Distractors

    • Reasoning of the Correct Response

    • Location of Relevant Information in Text

    • Length of Passage

    • Special Item Format

    Full item difficulty model

    • Propositional Density

    • Argument Structure

    • Text Length

    • Vocabulary Level

    • Sentence Length

    • Semantic Overlap

    • Level of Question

    • Vocabulary Level of Key and Distractors

    • Falsifiability of Distractors

    • Confirmation of the Key

    Full Item Difficulty Model

    Implications for score meaning
    Implications for Score Meaning

    • Variables related to both text encoding and decision processes were significant predictors in models of GRE-V item difficulty.

    • This suggests that GRE-V RC items measure both processes.

      • New evidence on construct validity of test scores for these items.

    Experimental manipulations
    Experimental Manipulations

    • Experimental conditions corresponded to variations in item features based on a hypothesized cognitive model.

      • Passage propositional density and syntax modification.

      • Passage passive voice and negative wording modification.

      • Passage order of information change.

      • Response alternative-passage overlap change.

    • Experimental effects tested with the LLTM model.

      • Rasch model to deconstruct sources of item difficulty.

    Implications for test design
    Implications for Test Design the Experimental Model

    • Significant effect of passive voice and negative wording on item difficulty was found.

      • Test writers often avoid the use of negative wording, citing that it is complicated and can confuse readers.

    • Although the effects were not significant for item difficulty, two significant effects on response time were found.

    Self report measures
    Self Report Measures the Experimental Model

    • Verbal Protocols

      • Concurrent or retrospective

    • Structured Questionnaires

      • Strategy use

      • Background information

      • Interest

      • Confidence

    Digital eye tracking
    Digital Eye Tracking the Experimental Model

    • Digital eye tracking data has been used to examine cognition and individual differences in

      • language processing.

      • facial processing.

      • learned attention.

      • electrical circuit troubleshooting.

      • problem solving strategies.

        • Spatial reasoning

        • Abstract reasoning

    The role of cognition in educational assessment design

    Look Zone 1 the Experimental Model

    Look Zone 2

    Look Zone 3

    The role of cognition in educational assessment design

    Look Zone 1 the Experimental Model

    Look Zone 2

    Look Zone 3

    Gazetrail for a reading comprehension item
    Gazetrail for a Reading Comprehension Item the Experimental Model

    • Question (0.797)

    • RO (0.297)

    • Passage (4.03)

    • Question (0.31)

    • Passage (41.35)

    • Question (2.59)

    • Passage (2.65)

    • RO (6.00)

    • Passage (3.09)

    • RO (0.76)

    • Question (0.37)

    • RO (.25)

    Point of gaze by time
    Point of Gaze by Time the Experimental Model

    Implications and future use
    Implications and future use. the Experimental Model

    • Verify some of our current models.

    • Identify new variables related to processing.

    • Describe qualitative differences characterized by strategy differences.

    • Examine specific aspects of test items that are problematic for individuals or for subgroups.

    • Observe the effects of controlled item manipulations on item processing, not just item responses alone.

    Summary of the benefits of cognitively based assessment design
    Summary of the Benefits of Cognitively-Based Assessment Design

    • Construct validity is more completely understood.

      • Explicitly elaborates the processes, strategies and knowledge structures.

    • Enhanced score interpretations.

      • Persons, as well as items, can be described by processes, strategies and knowledge structures.

    • Generation of items with specified sources and levels of item difficulty.

      • Item parameters may be predicted for newly developed items.

      • Items can be generated for specific populations by controlling the cognitive processing requirement.

    Greatest challenges
    Greatest Challenges Design

    • Our limited understanding of the cognitive models and the test items.

      • Current item response data is limited in information.

    • A “one size fits all” approach to cognitive modeling will not work.

      • Changing your goals for the assessment necessitates a change in the items.

      • Changing the items (or features of the items) implies a change in what can be concluded from the test.