Moving From Challenge To Action: Accountability Supporting Student Learning

1. Moving From Challenge To Action:Accountability Supporting Student Learning Joan L. Herman UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST) CRESST Conference UCLA, Los Angeles, CA September 9, 2004

2. Overview • Role of assessment in improving student learning • Evidence on how things are working • Advice on moving ahead

3. Theory of Action: Motivation • Establish standards • Develop measures • Set performance goals • Leaven with incentives/sanctions • Schools will be pay attention • Attend to standards/assessment results • Work to improve student performance/learning

4. Theory of Action: Technical System • Assessment results will provide accurate/valid information at multiple levels • Data will be well used to inform planning and decision making • Educational system will well use data to engage in continuous improvement

5. Is the Motivation System Working? • The good news: • Districts and schools taking action to align curriculum, etc with standards • Teachers listen to the signal and focus instruction accordingly • The not so good news • Curriculum narrowed • Possibility of dual curriculum • Concern for teacher’/principals’ professionalism and morale

6. Some Critical Questions: • How can we design standards and assessment systems that optimally focus instruction but don’t stifle it? • What happens to motivation when schools hit the wall and performance levels out?

7. Is the Technical System Working? • Feds are asking for right kinds of validity evidence • Alignment • Validity for EL and SWD • Accuracy: 75% probability that CST correctly classifies students’ proficiency level (Rogosa) • Reliability and year to year score fluctuations a continuing challenge

8. Is the Technical System Working?(cont.) • Validity of gains suspect • Feasibility of AYP targets problematic • Thorny technical/psychometric issues in dealing with 40+ possible indicators of AYP

9. Critical Questions • Validity of AYP designations: Do they identify the right schools? • Is there a better way to establish and operationalize improvement targets?

10. Alignment: A Critical Lynchpin • Are we clear enough about what we want: • Teachers to teach? • Test developers to test? • Is face validity of content and cognitive demand enough? • Research suggests difficulties

11. One Alignment Example • Study used 20 content experts • Considered each item and rated • Topic addressed • Depth of knowledge (DOK) • Content centrality • Results - Clear majority agreed on • Topic for 35/42 items • Topic AND DOK for 14/42 items • Topic, DOK and centrality for 5 items

12. Implications • How can teachers teach to “standards” if experts don’t agree on what topics and levels of DOK mean? • Current reality: alignment is a moving target

13. A Moving Target Example from NY(Alan Tucker) • Standard: Apply formulas to find measures such as length, area, volume, weight, time, and angle in real world situations • Representation of Math A Test, June, 02 - June, 03 (4 tests) • # items: 1-3 • Range of content and complexity

14. Critical Questions • If we want teachers to teach to the standards rather than the test, how can we be sure that developers and teachers share the same understandings? • How can we specify standards and assessments in ways that clearly communicate expectations without unacceptable curriculum narrowing? • Can we frame expectations in ways that are feasible for all students?

15. Teacher Assessment: The Neglected Key • If we believe in the power of assessment to support learning, we must put that power where learning occurs • Many capacity challenges • Some realistic remedies: • Teacher pre-service education • Requirements for materials developers

16. A Full Agenda for Deliberation and Action