Outcome-based and competition-based policies of school evaluation: a comparison of school performance (and perverse effe

1. Outcome-based and competition-based policies of school evaluation:a comparison of school performance (and perverse effects) in two mid-sized Hungarian towns HORN, Daniel BODIS, Lajos EUI, Florence Corvinus, Budapest daniel.horn@eui.eu bodis@uni-corvinus.hu Presentation at the INVALSI-APPAM “Improving Education through Accountability and Evaluation Lessons from Around the World” conference Rome, Italy, October 3-5, 2012.

2. Outline • National background • “Parktown” and “Birdtown” • outcome-based and competition-based policies of school evaluation • Quantitative results (empirics not presented) • Qualitative speculations • Conclusion Horn-Bodis - Outcome and Competition Based Policies

3. National Background • (practically) Free school choice • Per-student lump-sum grant • Demographic decline • Teachers arepublic servants (hard to fire, fixed salary scale) • Result: segregated school system (both student and teacher) (empirics for Hungary: Kertesi-Kezdi 2010, Varga 2011) Gradualadaptation: „competition-based policy” (Birdtown) • National Assessment of Basic Competencies (NABC) since 2001 • Schoolassessmentsystem (individual panel since 2010) Reform: „outcome-based policy” (Parktown) Horn-Bodis - Outcome and Competition Based Policies

4. The two towns • Birdtown • 75,000 inhabitants (county capital) • 2004: 14 schools – 7500 students • 2008: 11 schools – 6200 students (7,5% drop) • Parktown • 50,000 inhabitants (not a county capital) • 2004: 10 schools – 4100 students • 2008: 5 schools – 3500 students (4,7% drop) Horn-Bodis - Outcome and Competition Based Policies

5. Reform in Parktown • On the1st July 2007. • Restructuring the whole system • Closing all 10 schools and opening 5 new (2 closed, 2 merged into 2 other, and 1 handed over to the Church) • Firing all and hiring 70% of teachers (performance based) • Redrawing (and starting to use) catchment-areas • NABC was a tool in the reform (see below) Horn-Bodis - Outcome and Competition Based Policies

6. School performance - Birdtown • Focus on “quality” • Any school can be good in one (or more) aspects. • Testing is important (but not that much) • School evaluations are based on “constructive way of life”, “local patriotism” and “continuous professional development” “It was around 2005 when the politicians and administrators here [in Birdtown] started to say that the parents are voting with their feet. So we didn’t try to stop it. We said that if you can’t add a Chinese gymnastic course on top of ‘hottentot’ language course, and you can’t make them stay, we’re sorry.” (an administrator at the local government, Birdtown) Horn-Bodis - Outcome and Competition Based Policies

7. School performance - Parktown • Focus on standardized test-scores • Schools are compared on level scores annually • No-high stakes • Depends much on the (politically well imbedded and charismatic) mayor, who used the testing to hammer through his reform in 2006 “Then we went through each school one-by-one, which created the first smaller turbulence. They [the General Assembly] did not like it. They did not like to face the fact that there was practically no school that showed above national average performance. Thus we had no good schools. We had no exceptional primary schools.” (the mayor of Parktown) Horn-Bodis - Outcome and Competition Based Policies

8. Quantitative results • National Assessment of Basic Competencies (NABC) • School assessment based on standardized student test scores • Covering 6th, 8th and 10th grade • Math and reading • Since 2001, but panel since 2008-2010 • Detailed background questionnaire • Administrative dataset on all students • Panel data on 22 mid-sized towns between 2004-2006, 6th and 8th grade • Results (diff-in-diff): • Parktown does not differ from the other towns in level. (For Birdtown it is lower) • Value-added (VA) for Birdtown is non-significantly different from the other 20 towns • VA for Parktown is lower(!) after the reform Horn-Bodis - Outcome and Competition Based Policies

9. Qualitative speculations • Why have the VA dropped? • Bad (?) outcomemeasure: focus on level results • Teachers canplay the system • Teach to the test (we saw that) • Reallocate resources (this too) • Selective testing (have not documented) But no increase in level either! (not presented here) And the question is still valid: Why has VA dropped? Horn-Bodis - Outcome and Competition Based Policies

10. Qualitative speculations • Why has the VA dropped? • We are too early to judge (cf. Elmore 2003). Well, yes. • There are important missing pieces in the reform in Parktown. Horn-Bodis - Outcome and Competition Based Policies

11. Missing pieces • Introducing outcome regulation, inevitably puts a limit on the input as well as the process regulation (cf. Milgrom and Roberts 1992). • Parktown has changed input regulation. • but have not dealt with process measures: • Lack of „expert” teachers • Lack of forums for deliberation and feedback • Lack of (uniformly) good teacher training (participation) Horn-Bodis - Outcome and Competition Based Policies

12. Concluding thoughts • Doing a full-scale reform in a decentralized system has its benefits: • decrease between school segregation • fame within the country • but its effects on performance are questionable Horn-Bodis - Outcome and Competition Based Policies

13. Thank you for the attention! daniel.horn@eui.eu Horn-Bodis - Outcome and Competition Based Policies