Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council

1. Value-Added SystemsPresentation to the ISBE Performance Evaluation Advisory Council Dr. Robert H. Meyer Research Professor and Director Value-Added Research Center University of Wisconsin-Madison February 25, 2011

2. Attainment and Gain • Attainment – a “point in time” measure of student proficiency • compares the measured proficiency rate with a predefined proficiency goal. • Gain –measures average gain in student scores from one year to the next

3. Attainment versus Gain Gain Gain Gain Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

4. Growth: Starting Point Matters Reading results of a cohort of students at two schools Grade 4 Proficient Cutoff 438 Grade 5 Proficient Cutoff 463 *Scale Score Average is below Proficient Example assumes beginning of year testing

5. Value-Added • A kind of growth model that measures the contribution of schooling to student performance on the standardized tests • Uses statistical techniques to separate the impact of schooling from other factors that may influence growth • Focuses on how much students improve on the tests from one year to the next as measured in scale score points

6. Value-Added Model Definition • A value-added model (VAM) is a quasi-experimental statistical model that yields estimates of the contribution of schools, classrooms, teachers, or other educational units to student achievement, controlling for non-school sources of student achievement growth, including prior student achievement and student and family characteristics. • A VAM produces estimates of productivity under the counterfactual assumption that all schools serve the same group of students. This facilitates apples-to-apples school comparisons rather than apples-to-oranges comparisons. • The objective is to facilitate valid and fair comparisons of productivity with respect to student outcomes, given that schools may serve very different student populations.

7. A More Transparent (and Useful) Definition of VA • Value-added productivity is the difference between actual student achievement and predicted student achievement. • Or, value-added productivity is the difference between actual student achievement and the average achievement of a comparable group of students (where comparability is defined by a set of characteristics such a prior achievement, poverty and ELL status).

8. In English Post-on-Pre Link x Pretest Posttest = Student Characteristics School Effects Unobserved Factors + + + Value Added

9. VARC Philosophy • Development and implementation of a value-added system should be structured as a continuous improvement process that allows for full participation of stakeholders • Model Co-Build; Complete customization • Analysis • Reporting • Value–added is one tool in a toolbox with multiple indicators

10. VARC Value-Added Partners • Design of Wisconsin State Value-Added System (1989) • Minneapolis (1992) • Milwaukee (1996) • Madison (2008) • Wisconsin Value-Added System (2009) • Milwaukee Area Public and Private Schools (2009) • Racine (2009) • Chicago (2006) • Department of Education: Teacher Incentive Fund (TIF) (2006 and 2010) • New York City (2009) • Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts (2009) • Illinois (2010) • Hillsborough County , FL (2010) • Broward County, FL (2010) • Atlanta (2010) • Los Angeles (2010) • Tulsa (2010)

11. Districts and States working with VARC Minneapolis Milwaukee Madison Racine Chicago New York City Los Angeles Tulsa Atlanta Hillsborough County Broward County

12. Measuring knowledge • Many factors influence what a student learns and how their knowledge is measured • A variety of measures, including (but not limited to) assessments, tell us what a student knows at a point in time. • What are some ways we measure knowledge?

13. Measuring knowledge End-of-course Exam Diagnostic Test MAP WKCE Daily Journal Unit Project After-school Activities Hands-on Project

14. The Simple Logic of Value-Added Analysis • School Value-Added Report • School specific data • Grade level value-added • Comparison Value-Added Reports • Compare a school to other schools in the district, CESA, or state • Also allows for grade level comparisons • Tabular Data available for School Report and Comparison Reports

17. Attainment and Value-Added

18. How complex should a value-added model be? • Rule: "Simpler is better, unless it is wrong.“ • Implies need for “quality of indicator/ quality of model” diagnostics.

19. Model Features • Demographics • Posttest on pretest link • Measurement error • Student mobility: dose model • Classroom vs. teacher: unit vs. agent • Differential effects • Selection bias mitigation: longitudinal data • Test property analysis

20. MAP vs. ISAT • MAP dates: September, January, May • MAP: uses Rasch equating • ISAT: 3PL • MAP: slightly higher reliability - ~0.96 in math, ~0.94 in reading • ISAT math ~0.93, reading ~0.9 • Cut scores on MAP are determined by equipercentile equating to ISAT

35. Minimal correlation between initial status and value-added

39. Grade-Level Statewide Results

40. Grade-Level Statewide Results

41. Grade-Level Statewide Results

42. MPS and MMSD Value-Added compared to Wisconsin 6th to 7th Grade (Nov 2006 – Nov 2007) Mathematics – State VA Model School Effects MPS School Effects MMSD School Effects School/District VA Productivity Parameters in WKCE Scale Score Units (Relative to State)

43. Visit the VARC Website http://varc.wceruw.org/ for more information about VARC and value-added