Symposium: Assessment, Accountability, Instruction, and Learning in Urban Districts

1. Symposium: Assessment, Accountability, Instruction, and Learning in Urban Districts • Research funded by the Joyce Foundation and Helen Bader Foundation • Center for Systemic Reform in the Milwaukee Public Schools (SSR-MPS) • Wisconsin Center for Education Research (WCER) • University of Wisconsin-Madison • 3 papers: • - William Clune, et al. Milwaukee Middle School Proficiencies • - Robert Meyer Value-added & School Performance • - Norman Webb Assessment Literacy • All focus on Milwaukee district • 3 Commentators: • - Warren Chapman Joyce Foundation • - Deborah Lindsey Milwaukee Public Schools • - Andrew Porter WCER • Papers on website: http://www.wcer.wisc.edu/mps

2. The Milwaukee Middle School Proficiencies: Systemic school reform through high stakes assessments and a network of schools • William H. Clune, with Sarah Mason, Cecilia Pohs, • Chris Thiel, and Paula A. White • Paper prepared for the annual meeting of the AERA, • New Orleans, April 2, 2002 • Paper on website: http://www.wcer.wisc.edu/mps

3. Overview of Points • What are the Proficiencies? • Research Methods • Impact on instruction and achievement • Centralized/ decentralized policy & implementation • Value of standardized and performance assessments • Importance of evaluation

4. The Milwaukee Middle School Proficiencies: • What are they? • Promotion requirements from 8th grade, year • 1999-2000 through 2001-02 • 4 Areas: Communications, Mathematics, Science, • Research Paper • Multiple assessments, multiple opportunities to pass, • grades 6-8 • Purpose: assure readiness for high school (not just • pass or fail)

5. Assessment Types • Traditional standardized tests (e.g., State of Wisconsin) • On-demand district performance assessments • Performance assessments embedded in instruction • (tasks, scoring rubrics) • Most weight put on embedded assessments • Can pass w/o "proficient" on State test • Alternative completion project (all 4 proficiencies) • (Adopt-a-City)

6. Research Methods • 17 interviews in Spring 2000 • 11 school sites (range of reputed success) • 9 interviews with learning coordinators, 3 principals • 5 with respondents from school network ("Middle School Collaborative") • Taped, transcribed, coded with NUD*IST 4 • Names & schools not disclosed in paper

7. Findings • Strong impact on instruction, learning, school • organization • Unclear impact on student achievement, weak • positive evidence • Centralized/ decentralized implementation/ • policy formation

8. Finding 1: Strong Impact on Instruction, Learning, School Organization • Instruction •  Provided a focus for teaching (9 respondents) • Students re-do work w teachers (4) • Increased hands-on work (4) • Aligned with curriculum (9) • Learning •  Students: • - take greater responsibility (6) • - improved in writing (5) • - improved in reading (3) • - improved in math/ science (6) • - improved on standardized tests (2) • School organization •  Schools did major re-organization (4) • Special proficiency classes helpful (4) • Summer school programs helpful (3)

9. Finding 2: Unclear Impact on Student Achievement, weak positive evidence • Intended to improve achievement (proficiency beyond assessments) • Performance assessments not statistically reliable • Annual 8th grade testing poor method of evaluation • Wisconsin 8th grade scores rose for two years then fell • Technical problems: change of test date, test forms • District study: increase in high school grades and 9th grade promotion • Also more transitional ("8-T") students in grades 8-9 • Much better: value-added from annual standardized tests (Meyer paper)

10. Finding 3: Centralized/ decentralized implementation/ policy formation • In general • Unfolding requirements and implementation (incremental "roll out") • Much formative activity by District staff, Middle School Collaborative, • Learning Coordinators, Lead Principals • The Middle School Collaborative (network of Middle School Principals) • Had independent funding (Danforth) • Summer retreat for guiding vision ("all children can succeed") • Prevented repeal of Proficiencies • Successfully advocated alternative completion mechanism (Adopt-a-City) • Advocated fewer midstream policy changes

11. Discussion/ Significance of Findings • Policy strength from the top and bottom • Tradeoff of measurement reliability and • instructional validity

12. Discussion point 1: Policy strength from the • top and bottom • Good fit with Porter et al (1988) framework (authority, power, consistency, • specificity) • Importance of infrastructure in systemic reform (e.g., Clune, 2001) • At the top • - Authority (School Board, broad support for performance assessments) • - Power (promotion for students, high resources) • - Specificity (clear expectations for students) • - Consistency (coherent design across subjects, grades) • From the bottom • - Authority (Principals in Collaborative) • - Power (huge voluntary resources for implementation) • - Specificity (many details worked out between and w/i schools) • - Consistency (Vision and details managed decentrally)

13. Discussion point 2: Tradeoff Of Measurement Reliability And Instructional Validity • Traditional standardized tests: high reliability, low • instructional validity • District performance assessments: low reliability, high • instructional validity • Standardized performance assessments rejected as too expensive • Two good options for district: • - Combination of both, but move to annual standardized testing for evaluation • - Revisit standardized performance assessments

14. Conclusion: Importance of Evaluation • Important for both successful and unsuccessful • programs • To discontinue, continue, refine • Proficiencies used enormous resources (mostly labor) • Intensive resources = political vulnerability = need for • good evaluation