The Role of Cognition in Educational Assessment Design

1. The Role of Cognition in Educational Assessment Design Joanna S. Gorin Arizona State University

2. Overview • The need for cognitively-based assessment. • Defining cognitive models and their properties. • Tools for cognitively-based assessment design and analysis. • Empirical research on cognitive approaches to test design and validation.

3. The Purpose of Educational Assessment • New purposes for testing have introduced issues related to the inappropriateness of current standardized tests. Such assessments shall produce individual student interpretive, descriptive, and diagnostic reports…that allow parents, teachers, and principals to understand and address the specific academic needs of students, and include information regarding achievement on academic assessments aligned with State academic achievement standards, and that are provided to parents, teachers, and principals as soon as is practicably possible after the assessment is given, in an understandable and uniform format, and to the extent practicable, in a language that parents can understand. NCLB Part A Subpart 1 Sec. 2221(b)3(C)(xii), 2001

4. Implications of new assessment needs. • “[a]ll assessments will be more fruitful when based on an understanding of cognition in the domain and on the precept of reasoning with evidence” (NRC, 2001, p. 178) • Increase our understanding of the claims that we want to make about students, instruction, programs, and policy. • Detailed description of the skills, abilities, constructs we are measuring. • Increased understanding of the evidence provided by the student responses to the test questions. • Detailed description of item difficulty, discrimination, and other statistical and psychological properties.

5. Cognitively Based Assessment Design • If we begin with a complete understanding of the skill, domain, competency, ability, etc. that we want to measure, then we can be more principled in our design of the tools we use to measure it. • Further, if we understand our measurement tools and the scores they generate more fully, we can evaluate their quality relative to our measurement goals.

6. Cognitive Models and Grain Size • What is a complete understanding of the construct? Of the test? • Types of models: • Model of test specifications (content standards). • Model of task performance (cognitive processes). • Alignment of the test with standards or test specifications may not be sufficient to yield the desired information. • The model and alignment must be made at the appropriate level of inference.

7. Construct / Latent Trait Process A Process B Process C Process D Feature A Feature B Feature C Feature D Item / Test Question General Model of Cognitively-Based Assessment Design

9. Potential Cognitive Models • Information processing models. • Coherence-integration theory of text processing. • Propositional representation of text. • Cyclic construction of representation and integration of new information. • Activation theory. • Activation of information is influenced by various factors. • The most highly activated information will be selected. • Evaluate the “completeness” of these models in terms of the target inferences to be made about students.

10. Steps in Assessment Design Construct-based Model

11. Tools for Cognitively-Based Assessment Development • Design Frameworks • Mislevy’s Evidence Centered Design • A multi-level model of assessment. • Embretson’s Cognitive Design System • A process-based approach. • Bennett and Bejar’s Generative Approach • A structural approach.

12. Cognitively-Based Assessment Design Task-based Model

13. Purpose of Item Difficulty Modeling (IDM) • Extend available student and item information beyond a single statistical parameter. • Substantive information can be useful for: • Verification of construct definition (Construct Validity). • Creating new items (Automatic/Algorithmic Item Generation). • Providing diagnostic information (Score Reporting). • Understanding group differences (DIF).

14. Tools for Cognitively-Based Assessment Development • Design Frameworks for Test Construction • Evidence Centered Design • Cognitive Design System • Generative Approach • Psychometric Models for IDM • Tree Based Regression Approach • Rule Space Methodology • Attribute Hierarchy Method • Fusion Model • Linear Logistic Latent Trait Model

16. Traditional item-construct definition. • Reading comprehension questions measure the ability to read with understanding, insight and discrimination. This type of question explores the ability to analyze a written passage from several perspectives. These include the ability to recognize both explicitly stated elements in the passage and assumptions underlying statements or arguments in the passage as well as the implications of those statements or arguments.

17. Model of Test-Specifications (Standards and Objectives)

18. Cognitive Model Development • Cognitive Theory • Generate list of relevant processing components from theory. • Correlational Studies • Establish a statistical relationship between the features and the item properties. • Experimental Manipulations • Context/format • Item Design • Use process tracing methods to identify additional processing influences. • Verbal protocols (“think alouds”) • Eye-tracking data

19. Cognitive Model (Skill/Subskill Model)finer grain size

20. Cognitive Model (Information Processing Model)

21. Cognitive Variables • Modifier Propositional Density • Predicate Propositional Density • Text Content Vocabulary Level • Percent Content Words • Percent of Relevant Text • Falsification • Confirmation • Vocabulary Level of the Distractors • Vocabulary Level of the Correct Response • Reasoning of the Distractors • Reasoning of the Correct Response • Location of Relevant Information in Text • Length of Passage • Special Item Format

22. Vocabulary Level • Sentence Length • Propositional Density • Argument Structure • Text Length • Vocabulary Level • Sentence Length • Semantic Overlap • Level of Question • Vocabulary Level of Key and Distractors • Falsifiability of Distractors • Confirmation of the Key Full Item Difficulty Model

23. Activation Model of Item Difficulty

24. Regression Model of GRE Items

25. Contribution of Processing Factors to Item Difficulty

26. Implications for Score Meaning • Variables related to both text encoding and decision processes were significant predictors in models of GRE-V item difficulty. • This suggests that GRE-V RC items measure both processes. • New evidence on construct validity of test scores for these items.

27. Experimental Manipulations • Experimental conditions corresponded to variations in item features based on a hypothesized cognitive model. • Passage propositional density and syntax modification. • Passage passive voice and negative wording modification. • Passage order of information change. • Response alternative-passage overlap change. • Experimental effects tested with the LLTM model. • Rasch model to deconstruct sources of item difficulty.

28. Contrast Coding for LLTM Analysis

29. Scatterplot of Known and Estimated Item Difficulty Parameters for 29-Original

30. Effects of Manipulations on Item Difficulty

31. Table of Regression Coefficients and Significance Tests for the Experimental Model

32. Implications for Test Design • Significant effect of passive voice and negative wording on item difficulty was found. • Test writers often avoid the use of negative wording, citing that it is complicated and can confuse readers. • Although the effects were not significant for item difficulty, two significant effects on response time were found.

33. Self Report Measures • Verbal Protocols • Concurrent or retrospective • Structured Questionnaires • Strategy use • Background information • Interest • Confidence

34. Digital Eye Tracking • Digital eye tracking data has been used to examine cognition and individual differences in • language processing. • facial processing. • learned attention. • electrical circuit troubleshooting. • problem solving strategies. • Spatial reasoning • Abstract reasoning

35. Look Zone 1 Look Zone 2 Look Zone 3

36. Look Zone 1 Look Zone 2 Look Zone 3

37. Gazetrail for a Reading Comprehension Item • Question (0.797) • RO (0.297) • Passage (4.03) • Question (0.31) • Passage (41.35) • Question (2.59) • Passage (2.65) • RO (6.00) • Passage (3.09) • RO (0.76) • Question (0.37) • RO (.25)

38. Summary Data for Reading Comprehension Item

39. Individual vs Group Differences in Item Processing

40. Point of Gaze by Time

41. Implications and future use. • Verify some of our current models. • Identify new variables related to processing. • Describe qualitative differences characterized by strategy differences. • Examine specific aspects of test items that are problematic for individuals or for subgroups. • Observe the effects of controlled item manipulations on item processing, not just item responses alone.

42. Summary of the Benefits of Cognitively-Based Assessment Design • Construct validity is more completely understood. • Explicitly elaborates the processes, strategies and knowledge structures. • Enhanced score interpretations. • Persons, as well as items, can be described by processes, strategies and knowledge structures. • Generation of items with specified sources and levels of item difficulty. • Item parameters may be predicted for newly developed items. • Items can be generated for specific populations by controlling the cognitive processing requirement.

43. Greatest Challenges • Our limited understanding of the cognitive models and the test items. • Current item response data is limited in information. • A “one size fits all” approach to cognitive modeling will not work. • Changing your goals for the assessment necessitates a change in the items. • Changing the items (or features of the items) implies a change in what can be concluded from the test.