1 / 19

Estimating Performance Below the National Level

Fourth Annual IES Research Conference. Estimating Performance Below the National Level. Applying Simulation Methods to TIMSS. Dan Sherman, Ph.D. American Institutes for Research June 8, 2009. Current TIMSS Not Appropriate For Direct State-level Estimates.

nike
Download Presentation

Estimating Performance Below the National Level

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fourth Annual IES Research Conference Estimating Performance Below the National Level Applying Simulation Methods to TIMSS Dan Sherman, Ph.D. American Institutes for Research June 8, 2009

  2. Current TIMSS Not Appropriate For Direct State-level Estimates • Sample is designed to produce national rather than state estimates • Small number of schools sampled in about 40 states in 2007 • 234 public schools with 4th grade scores • CA has 30+ schools, FL, TX , and NY have 10+ schools • 207 public schools with 8th grade scores • CA has 20+ schools, TX, NY, and MI have 10+ schools • Large variations in school means within states makes direct estimates sensitive to choice of schools

  3. One Alternative to Direct Estimation for States Is to Use Regression Model Approach • Relate individual school-level mean score to variables observed for school, district, and state • Regression can take account of data structure(e.g., clustering of schools) in estimation • Use regression coefficients to create expected score for schools outside TIMSS sample • “Add up” expected scores across schools (weighted by number of students within state) • Provides estimate of mean score for state • Variance estimation is analytically complex; here handled by simulation

  4. Overview of Regression Model • Model begins with Common Core of Data (CCD) variables on right hand side • Share of students by gender, race, poverty status; also school size and location indicator (i.e.,urban , rural) • Can apply these variables to all public schools in sample • Then add average 2007 NAEP state math score for grade • Helps measure whether school in high or low performing state, relative to expectation of CCD variables • Also add in relative performance of school on state math test (percent proficient in school in SD units relative to mean percent proficient in state) • Helps adjust for position of school in state compared to other schools

  5. Goodness of Fit (Adjusted R2 ) for Alternative Specifications

  6. Summary of Alternative Regression Specifications • Models explain significant portion of variation in TIMSS (much more than PISA) • Demographics are (collectively) most significant group of variables • Poverty status is single best predictor variable • State NAEP score related to school mean, as is (strongly) position in state on test • Key is to predict large share of variance in mean scores of sampled schools • Larger R2 is desirable to closely track school mean estimates; otherwise will have large variance when school estimates aggregated to states

  7. Validation of School-level Model with (Three) Individual States • Can validate outside the model by predicting mean scores in 2 states with larger TIMSS samples (MA and MN with approximately 50 schools per grade per state • California design part of national sample • Fit (R2) between actual means and those predicted by model similar to overall model: • Obviously would like to validate/ examine model fit to schools in other states

  8. Illustration of Model Fit to Individual School Data – High R2

  9. Illustration of Model Fit to Individual School Data - Lower R2

  10. Approach to computing State-Level Means • Create estimate of school-level mean from TIMSS sample to create point estimate of mean for each school in CCD • Weight school estimates up to state-level mean using CCD number of students in grade • Should be unbiased in terms of expectation across repeated samples of schools

  11. Sources of Variance in Estimates of State-level Means • Mix and characteristics in TIMSS sample used in given regression • Different sample will produce different coefficients • Individual school estimate has prediction error around its expectation from regression • Standard error of regression – reduces with R2 and increases with distance from sample mean • Mean square error of regressions is about 20–25 points • Measurement error around mean (relatively small – about 5 points) for sampled schools

  12. Estimating Variance in State-Level Means • Take coefficients from regression model using 2007 TIMSS sample as given • Predict expected mean score for all regular public schools in CCD and add sources of error (i.e., random draws from distributions) for individual schools • Apply TIMSS sampling methodology (or any other!) and draw sample of schools • Estimate regression for this sample and compute coefficients • Compute state means from coefficient sample • REPEAT procedure drawing different samples to compute state-level means • Summarize means and standard deviation of estimated means (i.e, SEs) by state

  13. Illustration of National Results (4th Grade Math Scores)

  14. Summary of National Means and Standard Errors • Regression/ simulation provides national estimates similar to direct estimates • Can be “apportioned” to states pulling out observations for individual states

  15. Comparison of Model Results to Direct State-Level Estimates

  16. Some Observations on State-Level Estimates • 4th grade math presented as illustration; high correlation in state estimates across grade and subject (r = 0.90 to 0.98) • Median standard errors across states are 6.1 for 4th grade math, 3.3 for 4th grade science, 5.6 for 8th grade math, and 5.4 for 8th grade science • Large standard errors in HI , AK, and DC estimates reflect large population shares of minorities multiplied through by variance of coefficients of associated variables

  17. Potential Extensions • Current estimates are regression-based only and do not directly combine with existing state-level samples • Composite estimator would weight the two by relative variance, though current sample doesn’t support these estimators well with small state samples • Can validate model against two states (MA and MN) but would be useful to have other states with sufficient samples • Could work with 1999 data that had 13 state samples • Regression / Simulation approach could be used to assess alternative sampling schemes or models, given regression-based assumptions of scores at school level • Could draw larger samples to assess precision of direct estimates and compute composite estimates • Could draw from more high-minority schools to reduce standard errors for some states

  18. Conclusions • Regression model works well with current TIMSS given relatively high explanatory power of regression models • Direct estimates for most states would not be reliable/ credible with current sampling • Sample could be potentially modified to provide direct estimates for states and “borrow strength” from regression and require smaller state samples • Simulation can help evaluate sampling schemes a priori for complex estimators

More Related