Presented to: Oklahoma State Board of Education August 20, 2013

1. Oklahoma Interruption Investigation Arthur Thacker Presented to: Oklahoma State Board of Education August 20, 2013

2. Introduction to HumRRO • HumRRO is a 62 year old non-profit research company • Education clients include: • NAEP/NAGB • SMARTER Balanced • PARCC • State education agencies (OK, FL, KY, MN, ND, CA, PA, UT) • Services provided include: • Psychometric consulting and processing • Validity studies • Alignment studies • Standards setting • Quality assurance

3. Overview • Some students completing the OK assessments in spring 2013 experienced computer delays/interruptions. • The focal disruptions occurred on 4/29 and 4/30, although other disruptions were recorded on other days. • The methodology and interpretation of results was conducted independently of the testing contractor.

4. Overview (cont.) • The purpose of this investigation was to determine if computer disruptions affected student scores • Specifically, the concern lies in disrupted student scores being lower than expected • We investigated multiple groups of interrupted students (grades 6-8 and high school) using multiple methods in an attempt to “converge” on a bias, if any could be detected.

5. Challenges to the Investigation • Students were not interrupted randomly or by design • Computer interruptions are not that uncommon, even when there is no identified issue during testing to be discovered • Individual students may have responded very differently to the interruption

6. Structure • Four “Cohorts” were investigated • Cohort A – All students who had an interruption in the Grades 6-8 dataset, regardless of day • Cohort B – Only examined those with interruptions on 4/29 and 4/30 • Cohort C – EOI Data for Algebra scores • Cohort D – EOI Data for English scores

7. Propensity Score Matching • Propensity scores used to “mimic” an equivalent sample for comparison to the interrupted group • Matched the interrupted sample with individuals in the non-interrupted group who were similar on all available variables that relate to 2013 scores, including: • Prior year scores • School-level scores • Ethnicity • Gender • Free/Reduced Lunch

8. Algebra and English II EOI • Two sets of analyses were conducted for each EOI exam • Algebra • Grade 9 students in 2013 with Algebra scores matched to their Grade 8 Math scores from 2012 • Grade 7 and 8 students in 2013 with Algebra scores matched to their 2013 math (and reading) scores • English II • Grade 10 students in 2013 with English II scores matched to their 2013 Grade 10 US History scores • Grade 10 students in 2013 with English II scores matched to their 2011 Grade 8 reading (and math) scores

9. Analyses • All analyses performed on the matched Disrupted and Non-Disrupted groups • Mean differences on 2013 scores (Statistical, Meaningful) • R2 differences when predicting 2013 scores separately • R2 change when combining groups and adding dichotomous disruption variable • Applying Non-Disrupted regression equation from step 2 on Disrupted group as well as 5th, 10th, 90th, and 95th percentile cuts

10. Conclusion • The evidence shows that the effect of disruption on students’ scores was neither widespread nor large and the conclusions were not consistent across methods. • In addition, the results show that the effects did not always disadvantage the disrupted students, but at times the disrupted students did better than expected. • The most significant disruption impact seems to be for Algebra students in Grades 7 and 8, but the impact was relatively small and inconsistent across the distribution of students. Lower performing students tended not to perform as well as predicted.