1 / 61

Centre for Market and Public Organisation

Centre for Market and Public Organisation. A natural experiment in school accountability: the impact of school performance information on pupil progress and sorting Simon Burgess, Deborah Wilson and Jack Worth. Introduction. How to improve educational attainment?

emory
Download Presentation

Centre for Market and Public Organisation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Centre for Market and Public Organisation A natural experiment in school accountability: the impact of school performance information on pupil progress and sorting Simon Burgess, Deborah Wilson and Jack Worth

  2. Introduction • How to improve educational attainment? • And reduce educational inequalities? • Role of school accountability ... • System in England and Wales since 1988: • Market-based accountability • Administrative accountability • It is “consequential accountability” (Hanushek and Raymond) www.bris.ac.uk/cmpo

  3. Evaluating school accountability • Hard to get at a causal effect (Figlio and Ladd, 2008): • Lack of adequate control group • Introduction of multi-faceted system all at once. • Hanushek and Raymond (2005) and Dee and Jacob (2009) – evaluate the effects of NCLB • We exploit an event that gets round these problems: www.bris.ac.uk/cmpo

  4. From 1992 to 2001, school performance tables (“league tables”) published annually in England and Wales • Devolution of power to Welsh Assembly Government (WAG) after a referendum in 1999 • WAG abolished the publication of these league tables from 2001; they continued in England. • Otherwise, the educational systems continued to be very similar; some later changes in KS testing. • We set up a difference-in-difference approach. www.bris.ac.uk/cmpo

  5. What we do • Test whether the removal of this accountability mechanism: • Reduces school effectiveness • Reduces sorting across schools. • Why might these happen (or not)? • Principal agent model – performance tables provide public scrutiny of the output of the school • Teachers are professionals ... • Information as basis for sorting (for some?) www.bris.ac.uk/cmpo

  6. Results • Significant and sizeable negative effect on pupil progress: 2 GCSE grades = 0.23 sd (school) = 0.09 sd (pupil) • Equivalent to raising class-sizes from 30 to 38 • Heterogeneity: • Greatest effect on schools with most poor children • No effect in the top quartile of schools or schools with fewest poor children • No effect on sorting; if anything a hint of polarisation in Wales. www.bris.ac.uk/cmpo

  7. What is the mechanism? • Less information for choice? • No effect for top schools • But, little variation by degree of competition • Diminished government scrutiny? • Different culture and different emphasis of WAG • Diminished local public accountability? www.bris.ac.uk/cmpo

  8. Objections? • Schools in Wales have lower funding • We match schools and control for funding • Poorer neighbourhoods and families • We match schools and use school fixed effects • Other sources of information • We can’t find any, nor can The Times • “Gaming” and manipulation of qualifications • Our evidence does not support this; PISA findings • Confounded with other policies • No obvious ones fit with the timing and distribution www.bris.ac.uk/cmpo

  9. PISA test scores for England and Wales www.bris.ac.uk/cmpo

  10. Broader issues? • This is a specific but important question (HE, jobs, …) • Other issues: • Learning in a broader sense than GCSE performance • PISA evidence • Pupil well-being • … and GCSE performance • For its own sake • Teacher well-being • … and GCSE performance • For its own sake www.bris.ac.uk/cmpo

  11. Change of policy: • Leighton Andrews’ Speech, 2nd February, 2011, Cardiff. (Minister for Children, Education and Lifelong Learning): “We will introduce a national system for the grading of schools which will be operated by all local authorities/consortia. … All schools will produce an annual public profile containing performance information to a common format” www.bris.ac.uk/cmpo

  12. Plan • Introduction • Policy Environment and the Policy Change • Methodology • Data • Results • School effectiveness • Sorting • Conclusion www.bris.ac.uk/cmpo

  13. School system • State-funded schools = 94% students • National Curriculum, four Keystages • Primary Education, to age 11, compulsory secondary education to age 16. • Keystage 3 exams at age 14, Keystage 4 exams at age 16, also called GCSEs • GCSEs are high stakes exams for students – access to higher education and to jobs www.bris.ac.uk/cmpo

  14. School accountability • GCSE exams taken May/June, results reported privately to pupils in August, league tables published November. • Very widespread media attention for these: national and local tv, radio, newspapers ... • Key indicator is % students getting 5 or more “good passes” (grade C or higher). • Other main aspect of accountability: school inspections by OFSTED (E) and ESTYN (W) www.bris.ac.uk/cmpo

  15. Policy change in Wales • Referendum in Wales voted in favour of devolved government • Welsh Assembly Government set up in Cardiff, and took over education policy. • Announced in July 2001 the abolition of league tables, and they were not published in November 2001 in Wales, nor since. • Rest of system continued: National Curriculum, GCSEs, ... (some other aspects of testing changed later on). www.bris.ac.uk/cmpo

  16. Why? • Arguments against league tables ... • In Wales, perhaps largely ideological, and an explicit avowed preference for “producer” interests over a “consumerist” approach. • Reynolds (2008): reform was “motivated by the left wing political history of Wales ... use of government to ensure enhanced social justice .... greater trust in producer-determined solutions” www.bris.ac.uk/cmpo

  17. Potential off-setting changes 1? • Increased school inspections? • No, no change of policy by Estyn (WAG) • Schools publish own results? • Yes, some do; • We surveyed schools in Cardiff and (similar) Plymouth and Newcastle. • Low-performing schools tend not to, and are not obliged to (our survey); • Nothing overtly comparative in Wales. www.bris.ac.uk/cmpo

  18. Potential off-setting changes 2? • Other ad hoc private websites? • Not that we (or The Times) could find • Broader accountability system: • Nature and characteristics of choice systems have not changed differentially in the two countries • Top-down accountability: • school performance data continue to be analysed by local government in Wales “as a basis for challenge and discussion” www.bris.ac.uk/cmpo

  19. Methodology • Difference-in-difference, with: • school fixed effects • time varying controls • sample 1: all schools (balanced) • sample 2: matched schools • Two outcomes: • School %5+ good passes (published in league tables) • Mean GCSE points (not consistently published) www.bris.ac.uk/cmpo

  20. School is the appropriate unit • Usual assumptions: • Non-controlled factors have common time patterns • No change in population characteristics • These are discussed below www.bris.ac.uk/cmpo

  21. Matching schools • Take all secondary schools in Wales from balanced sample • Matching variables: performance (level and trend), composition (FSM, ethnicity), school resources, local competition, all averaged over the “before” period. • Match English schools by propensity score to one nearest neighbour in Wales, without replacement • Differences in the means of matching variables are individually and jointly insignificant. www.bris.ac.uk/cmpo

  22. Propensity Score Matching (School - Performance) www.bris.ac.uk/cmpo

  23. Propensity Score Matching (LEA - Sorting) www.bris.ac.uk/cmpo

  24. Data • NPD/PLASC • School performance • Data • Sample • Sorting • Data • Sample www.bris.ac.uk/cmpo

  25. Schools data • PLASC/NPD – Pupil Level Annual Schools Census, part of the National Pupil Database • School-year (cohort) level • GCSE scores (two different metrics) • Prior attainment, Keystage 3 scores (KS3) • Poverty measure – eligibility for free school meals (FSM) • School resources (annual expenditure outturn), deflated by CPI and by Area Cost Adjustment • %White British, % Female www.bris.ac.uk/cmpo

  26. Sample • Exclude all schools in areas > 10% selective • Wholly-before and wholly-after • Effectiveness (key ages 14 – 16): • Wholly before cohorts are A and B, taking GCSEs in 2000 and 2001; wholly after cohorts are E onwards, taking GCSEs from 2004 • Sorting (key ages 10 – 11): • Cohorts E – H are before, cohort I – ... after www.bris.ac.uk/cmpo

  27. Performance Timeline www.bris.ac.uk/cmpo

  28. Sorting Timeline www.bris.ac.uk/cmpo

  29. Results • Impact of league tables on performance • Simple diff-in-diff • Full diff-in-diff • Heterogeneity • Robustness • Impact of league tables on sorting • Dissimilarity indices • School poverty rates • School – Neighbourhood assignment www.bris.ac.uk/cmpo

  30. School Mean GCSE Points score in England and Wales over time www.bris.ac.uk/cmpo

  31. School Percent 5 A*-C in England and Wales over time www.bris.ac.uk/cmpo

  32. PISA scores www.bris.ac.uk/cmpo

  33. Simple Difference-in-differences www.bris.ac.uk/cmpo

  34. Full Difference-in-Difference Model Also included: school fixed effects, year effects, prior attainment. SE’s clustered at LA level www.bris.ac.uk/cmpo

  35. Results • Heterogeneity • Year • Quartiles of school prior attainment • Quartiles of school poverty • Quartiles of league table position • Quartiles of school local competition www.bris.ac.uk/cmpo

  36. Full D-in-D allowing for Heterogeneity through time www.bris.ac.uk/cmpo

  37. Full D-in-D allowing for Heterogeneity in Prior Attainment www.bris.ac.uk/cmpo

  38. Full D-in-D allowing for Heterogeneity in Poverty Status www.bris.ac.uk/cmpo

  39. Full D-in-D allowing for Heterogeneity in Local Competition www.bris.ac.uk/cmpo

  40. Results • Robustness • Serial correlation (Bertrand, Duflo and Mullainathan, 2004) – collapse time dimension • Changes in group composition: • Private schools • Cross-border movement • Other aspects of devolution ... • Coincident policy changes: • Literacy hour • Staggered introduction of new equivalent qualifications www.bris.ac.uk/cmpo

  41. Collapse to Pre- and Post-reform periods www.bris.ac.uk/cmpo

  42. Difference-in-difference of Composition Variables www.bris.ac.uk/cmpo

  43. Primary - Secondary Triple Difference www.bris.ac.uk/cmpo

  44. Coincident policy changes 1 • Literacy hour introduced in primary schools in England in 1998, so our first “after” cohort exposed in England and not Wales • Control for prior attainment; hard to see an age 5-11 policy affecting age 16 score, conditioning on age 14 ability • Effect only in urban areas, and most of matched sample is rural www.bris.ac.uk/cmpo

  45. Coincident policy changes 2 • Staggered introduction of replacement GCSE-equivalent qualifications, in England in 2005 and in Wales in 2007 • Minority activity: affects 4.3% of GCSE points at the median in England in 2006, only 9.3% in lowest decile • Would expect to see differential time trends off the effect over KS3 distribution, but not so. www.bris.ac.uk/cmpo

  46. D-in-D allowing for Heterogeneity in Prior Attainment and Through Time www.bris.ac.uk/cmpo

  47. Results • Impact of league tables on sorting • Dissimilarity indices, national averages over LAs • Figures • Diff-in-diff regressions • Differential evolution of school poverty rates • Figures • Diff-in-diff regressions • School – Neighbourhood assignment • Figures • Diff-in-diff regressions www.bris.ac.uk/cmpo

  48. LEA-level FSM Dissimilarity Index in England and Wales over time www.bris.ac.uk/cmpo

  49. Difference-in-Difference model: Dissimilarity Index: FSM www.bris.ac.uk/cmpo

  50. School poverty evolution • Match all Welsh schools to English schools on FSM % in the ‘before’ period • Split into quartiles • Trace out evolution of school FSM% quartile-by-quartile separately in England and Wales • Normalised by national trends • Poor schools less poor in Wales, affluent schools poorer?? www.bris.ac.uk/cmpo

More Related