1 / 17

What’s the Matter?

What’s the Matter?. Methodological Challenges in School Effectiveness Research. Session Objectives. Why is school effectiveness hard? Causality, complex structure, change, measurement ... What models/methods are available? Multi-level, structural equation, multiple membership, MCMC ...

freira
Download Presentation

What’s the Matter?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What’s the Matter? Methodological Challenges in School Effectiveness Research

  2. Session Objectives • Why is school effectiveness hard? • Causality, complex structure, change, measurement ... • What models/methods are available? • Multi-level, structural equation, multiple membership, MCMC ... • Conclusions • Evidence about school effectiveness, future direction Methodological Challenges for School Effectiveness

  3. Great Scientific Advances What ? How ? Why ? School Effectiveness (1964- ) The Double Helix (1944-1953) One Giant Step for Mankind (1961-1969) Coleman et al. ( 1966); (Strand 2007) Methodological Challenges for School Effectiveness

  4. School Performance is Dynamic • Parents want to know about future performance; school league tables record the past • Time-School-Pupil: a 3 level model • Effectiveness changes; so does the definition of “effective” Leckie and Goldstein (2009) Methodological Challenges for School Effectiveness

  5. School Pupils are Dynamic • The education system is dynamic • 43% of primary pupils move between schools • League tables credit achievement to last school • Needs multiple membership and cross-classification • Pro rata apportionment corrects school effects by 17% Creemers, Kyriakides and Sammons (2010); Goldstein, Burgess and McConnell (2007); Leckie (2008) Methodological Challenges for School Effectiveness

  6. Attainment and Attenuation • Prior attainment is hugely important in CVA • Standard 2-level models omit measurement error • Unaccounted measurement error attenuates the estimated slope at the pupil level Raudenbush and Bryk (2002); Woodhouse, Yang, Goldstein and Rasbash (1996); Hutchison (2007) Methodological Challenges for School Effectiveness

  7. Phantom Contextual Effects • The estimate at the school level is less attenuated • The CVA model has a different parameterization • Where • Differential attenuation = Phantom contextual effect Woodhouse, Yang, Goldstein and Rasbash (1996); Harker and Tymms (2004) Methodological Challenges for School Effectiveness

  8. Measurement and School Effects • School effects are assessed using variance components • Phantom effects carry through to residual variation • We under-estimate VPC unless we omit the context Methodological Challenges for School Effectiveness

  9. Poverty of Data • Low socio-economic status (SES) pupils make slower academic progress but SES is hard to collect • So DCSF uses free school meals (FSM) • FSM has a number of weaknesses • Only one component of SES • Dichotomous • Missing data • High mis-classification rates Gorard (2009); Kounali, Goldstein and Lauder (2007) Methodological Challenges for School Effectiveness

  10. Measuring Dichotomously Methodological Challenges for School Effectiveness

  11. Measuring Dichotomously ... Badly Expected Value Actual Observed 16/17.2 40% 17.2% 16% FSM=1 24% .64 1.2% 60% 82.8% FSM=0 24/82.8 58.8% Kounali, Goldstein and Lauder (2007); Goldstein, Kounali, and Robinson (2007) Methodological Challenges for School Effectiveness

  12. FSM and School Effects • Unreliability due to dichotomous measurement and mis-classification compound • CVA omits a contextual term for FSM • Only observing a fraction of FSM pupils may cause us to further overestimate school effects Goldstein, Kounali, and Robinson (2007) Methodological Challenges for School Effectiveness

  13. Manifestations of the Phantom • The phantom effect manifests on compositional variables • “... differences in the composition of schools’ intake achievements ... appears to be a major driver of between-school differences in GCSE scores.” • “school level coefficients which show strong differential effects for prior achievement, FSM and SEN.” Leckie and Goldstein (2009); Leckie (2008) Methodological Challenges for School Effectiveness

  14. Discriminating Among Schools • Discrimination between schools is (largely) a function of school size and VPC • Coleman (1966) estimated VPC at around 10% • “ ... adjusting for these [compositional] variables halves the between-school variance; the VPC drops from 10.4% to 7.2%” Leckie and Goldstein (2009) • To improve discrimination, we need to allow for measurement error Methodological Challenges for School Effectiveness

  15. Dealing with Unreliability • Methods to deal with unreliability need to : • Work in conjunction with complex, multi-level methods • Handle complex, multi-level error and categorical variables • Some competing methodologies: • Regression discontinuity • Multiple measures in MLSEM • Item Response Theory (IRT) • Sensitivity analysis approach (MCMC) Luyten (2006); Luyten and Tymms (2010 ); Marsh et al. (2009); Fox (2004 ); Ecob and Goldstein (1983); Goldstein, Kounali, and Robinson (2007) Methodological Challenges for School Effectiveness

  16. Summary / Conclusions • Phantom effects are ubiquitous in school effectiveness research • FSM is deeply flawed as a proxy for SES and makes interpretation of results very problematical • Merely including cluster-level aggregates will tend to understate school-level residual variance • SER needs complex ML methods in combination with sophisticated tools to model unreliability Methodological Challenges for School Effectiveness

  17. Bibliography Equality of Educational Opportunity (Coleman et al. 1966) Minority Ethnic Pupils in the Longitudinal Study of Young People in England (Strand 2007) The estimation of school effects (Raudenbush and Willms 1995) Methodological Advances in Educational Effectiveness Research (Creemers, Kyriakides and Sammons 2010) The limitations of using school league tables to inform school choice (Leckie and Goldstein 2009) Incorporating Student Mobility in Achievement Growth Modeling: A Cross-Classified Multiple Membership Growth Curve Model (Grady and Beretvas 2010) Modelling The Effect Of Pupil Mobility On School Differences In Educational Achievement (Goldstein, Burgess and McConnell 2007) Effects of Pupil Mobility and Neighbourhood on School Differences in Attainment (Leckie 2008) School Value Added Measures in England (Ray 2006) The Effects of Student Composition on School Outcomes (Harker and Tymms 2004) Modelling Response Error in School Effectiveness Research (Fox 2004) Multilevel, Fully Latent-variable Models of Contextual Effects ... (Marsh et al., 2009) Instrumental variable methods for the estimation of test score reliability (Ecob and Goldstein 1983) Modelling measurement errors and category misclassifications in multilevel models (Goldstein, Kounali, and Robinson 2007) An empirical assessment of the absolute effect of schooling: regression discontinuity applied to TIMSS-95 (Luyten 2006) The probity of free school meals as a proxy measure for disadvantage (Kounali, Goldstein and Lauder 2007) Modelling measurement errors and category misclassifications in multilevel models (Goldstein, Kounali, and Robinson 2007) Serious doubts about school effectiveness (Gorard 2009) Methodological Challenges for School Effectiveness

More Related