Knowing What Students Know

1. Knowing What Students Know Ganesh Padmanabhan 2/19/2004

2. Overview • Nature of Assessment • Need for Reform • Advances in Cognition • Advances in Measurement • Role of Technology

3. Assessment • “Assessment is the means used to measure the outcomes of education and achievement of students with regard to important competencies.” • Purpose determines priorities. • Context of use imposes constraints on design.

4. Purposes of Assessment • Achievement vs Aptitude • Formative vs Summative • Evaluation of Educational Programs • More and more high-stakes decisions • One type does not fit all • Need for better alignment

5. Reasoning From Evidence • “Assessment is a tool designed to observe students’ behavior and produce data that can be used to draw reasonable inferences about what the students know.” • Also inferences about how, when, and whether they use what they know. • Imprecise by-nature

6. Assessment Triangle Observation Interpretation Cognition

7. Cognition • “…refers to a theory or set of beliefs about how students represent knowledge and develop competence in a subject domain.” • Cognitive Theories • Educational Theories • Experience of Expert Teachers • Need not be exhaustively complex

8. Observation • “Every assessment is also based on a set of beliefs about the kinds of tasks or situations that will prompt students to say, do, or create something that demonstrates important knowledge and skills.” • Not arbitrary, needs careful design.

9. Interpretation • “…expresses how the observations derived from a set of assessment tasks constitute evidence about the knowledge and skills being assessed.” • Methods for drawing inferences • Classroom – usually qualitative • Large-Scale – usually formal, statistical

10. Making Sound Inferences • Depends on making explicit connections in assessment triangle between • Cognition and Observation • Cognition and Interpretation • Observation and Interpretation • Interdependent and Iterative Development

11. Advances in the Learning Sciences • “How People Learn” by NRC (1999) • Systematic investigation dates back to the late 19th Century • Recently, very diverse fields converging • “Cognitive Revolution” (1960s onward) • Four significant perspectives: Differential, Behaviorist, Cognitive, and Situative

12. Differential Perspective • “The differential perspective focuses mainly on the nature of individual differences in what people know and in their potential for learning.” • Early 1900s • Bell Curve Assumptions • Focus on aptitude not achievement

13. Behaviorist Perspective • Skills are thought to be composed of stimulus-response associations. • Considers learning as the process by which one acquires those associations and assembles them into skills. • Stimulus, reinforcement, conditioning theories etc.

14. Cognitive Perspective • “In cognitive theory, knowing means more than the accumulation of facts and knowledge; it means being able to integrate knowledge, skills, and procedures in ways that are useful for interpreting situations and solving problems.” • Aids assessment in figuring how, when, and whether students use what they know.

15. Situative Perspective • Sociocultural View • Places individual thinking within a practical context • Authentic Assessment • Needs more here.

16. Interpreting Evidence • Psychometrics • Probabilistic Approach to Reasoning • Cognition  Construct • Observation  Observational Model • Interpretation  Measurement Model

17. Student Construct Observations Generation of Data ө x Interpretation of Data “…statistical models can be developed to predict the probability that people will behave in certain ways in assessment situations, and that evidence derived from observing these behaviors can be used to draw inferences about students’ knowledge, skills, and strategies…”

18. Classical Test Theory Construct ө Sum of Responses Observations

19. Generalizability Theory ө Construct Sum of Responses Observations Raters Type of Task

20. Item Response Modeling ө Construct Response 1 Response 2 Response 3 Response 4 Response 5 Observations δ1 δ2 δ3 δ4 δ5

21. Latent Class Models • Fig 4-8

22. Multi-Attribute Models • Fig 4-9

23. Role of Technology • Theory-Based Item Generation • Concept Organization • Complex Problem Solving (e.g. IMMEX) • Text Analysis and Scoring • Portal computer-aided approach for design of assessment

24. Questions

25. References All quotes and figures were taken from the NRC Report “Knowing What Students Know” by Pellegrino et al. (2001)