Evaluating the Validity of Student Learning Objectives

1. Evaluating the Validity of Student Learning Objectives Katie Buckley, Harvard University Scott Marion, Center for Assessment National Conference on Student Assessment (NCSA) National Harbor, MD June 22, 2013

2. Overview of Presentation • Theory of Action for SLOs • Validity Argument for SLOs tied to ToA • Research Agenda based on Validity Argument Buckley & Marion. CCSSO 2013

3. What is an SLO? • Content- and grade/course-specific measurable learning objectives that can be used to document student learning by a teacher over a defined period of time • Designed to involve teachers throughout the process and incentivize good teaching practices • Can be used as both an accountability tool and as an instructional tool Buckley & Marion. CCSSO 2013

4. Why do we care? • SLOs currently used in teacher evaluation systems by for teachers in non-tested grades and subjects (NTSG) • Upwards of 20 states/districts are using SLOs • Nearly 70% of teachers teach in NTSG • However • Very little research currently exists • Much evidence is needed to substantiate the implicit and explicit claims to support the use of SLOs in educator evaluations Buckley & Marion. CCSSO 2013

5. Theory of Action as a Starting Point • A theory of action (ToA)is a useful way to structure a validity evaluation in complex contexts… • Researchers have suggested (and we believe) that a theory of action can*: • Frame the validity evaluation, • Serve as a useful starting point in creating a validity argument, • Address the program evaluation and consequential aspects of the system not easily addressed through a validity argument *(Bennett, 2010; Marion & Pellegrino, 2006; Marion, 2010; Ryan, 2002). Buckley & Marion. CCSSO 2013

6. The Challenges with Creating a ToA for SLOs • SLOs are more than an assessment or program • Student “growth” is embedded within the SLO process rather than measured externally • It is important to convey how teachers should be involved in the SLO process • SLOs are flexible and may be adapted to a variety of systems Buckley & Marion. CCSSO 2013

7. Components of an SLO • SLO is a framework that includes: • Meaningful learning expectations (goals) • High quality assessments • Targets for student performance • Targets for aggregate educator performance • Instruction from teachers • Therefore, an evaluation of SLOs must address all five components Buckley & Marion. CCSSO 2013

8. Theory of Action for SLOs in a Teacher Evaluation System Buckley & Marion. CCSSO 2013

9. Expanding the ToA with a Validity Argument • We expand the ToA to create a validity argument based on Kane’s 2006 work • We explicate for each condition/input: • Propositions • Claims • Example Evidence • Propositions are organized according to • Scoring • Generalization • Extrapolation • Decision • Consequence* Buckley & Marion. CCSSO 2013

10. Input 1: Student learning goal is rigorous yet attainable Buckley & Marion. CCSSO 2013

11. Input 2: Assessments accurately and reliably measure student learning of goals Buckley & Marion. CCSSO 2013

12. Input 3: Targets set for students are appropriate and valid Buckley & Marion. CCSSO 2013

13. Input 4: Targets set for teachers are appropriate and valid Buckley & Marion. CCSSO 2013

14. Input5: SLO is adequately tied to teacher instruction Buckley & Marion. CCSSO 2013

15. Prioritizing Studies • It would be impossible to conduct studies for each claim listed, so how to prioritize? • State’s chosen priorities • Analyses that provide monitoring results for system-wide improvement • Analyses that address multiple claims • Need to also consider short-term vs. long-term analyses • What can be done/should be done now based on available data vs. later when the system has produced more data? Buckley & Marion. CCSSO 2013

16. Research Agenda • Three Studies • 2 based on quantitative data currently available • 1 based on qualitative data not yet available • Will help to determine whether: • SLOs are being implemented as intended • Results that are produced are as hypothesized • Address multiple claims • Analyses that can be used by GA to respond to USED’s requests Buckley & Marion. CCSSO 2013

17. Research Agenda • Study 1: Are there differences in the quality and attainment of SLOs: • By district, grade, course, course-type? • Across school and classroom demographics/characteristics? • Over time? • Claims addressed • Goals are of high quality (Learning Goal input, Scoring proposition) • Student attainment of SLO is not due outside factors beyond the teacher’s control and influence (Teacher Target input, Decision proposition ) Buckley & Marion. CCSSO 2013

18. Research Agenda • Study 2: What is the relationship between attainment of SLOs and other measures of teacher effectiveness, including: • Student growth? • Teacher practice? • Claims Addressed: • Attainment of the learning goals can be used to classify teacher facilitation of student performance (Learning Goal Input, Decision proposition) • Student attainment of target provides evidence of overall student achievement (Student Target input, Extrapolation proposition) • Attainment of target provides evidence of overall teaching effectiveness(Teacher Target input, Extrapolation proposition) • Higher scores on SLOs reflects more focused teaching of important content frameworks (Teacher Instruction input, Decision proposition) Buckley & Marion. CCSSO 2013

19. Research Agenda • Study 3: How do teachers and school leaders understand the SLO system and use the system for instructional purposes? • Claims addressed: • Teachers modify/alter instruction in direct response to data indicating students are off target from meeting their SLO (Teacher Instruction input, Generalization proposition) • Higher scores on SLOs reflects more focused teaching of important content frameworks (Teacher Instruction input, Extrapolation proposition) Buckley & Marion. CCSSO 2013

20. Thank you • Questions/comments: • Katie Buckley (buckley.kate@gmail.com) • Scott Marion (smarion@nciea.org) Buckley & Marion. CCSSO 2013