slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Presented to: Oklahoma State Board of Education August 20, 2013 PowerPoint Presentation
Download Presentation
Presented to: Oklahoma State Board of Education August 20, 2013

Loading in 2 Seconds...

play fullscreen
1 / 10

Presented to: Oklahoma State Board of Education August 20, 2013 - PowerPoint PPT Presentation


  • 99 Views
  • Uploaded on

Oklahoma Interruption Investigation. Arthur Thacker. Presented to: Oklahoma State Board of Education August 20, 2013. Introduction to HumRRO. HumRRO is a 62 year old non-profit research company Education clients include: NAEP/NAGB SMARTER Balanced PARCC

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Presented to: Oklahoma State Board of Education August 20, 2013' - jase


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Oklahoma Interruption Investigation

Arthur Thacker

Presented to:

Oklahoma State Board of Education

August 20, 2013

introduction to humrro
Introduction to HumRRO
  • HumRRO is a 62 year old non-profit research company
  • Education clients include:
    • NAEP/NAGB
    • SMARTER Balanced
    • PARCC
    • State education agencies (OK, FL, KY, MN, ND, CA, PA, UT)
  • Services provided include:
    • Psychometric consulting and processing
    • Validity studies
    • Alignment studies
    • Standards setting
    • Quality assurance
overview
Overview
  • Some students completing the OK assessments in spring 2013 experienced computer delays/interruptions.
  • The focal disruptions occurred on 4/29 and 4/30, although other disruptions were recorded on other days.
  • The methodology and interpretation of results was conducted independently of the testing contractor.
overview cont
Overview (cont.)
  • The purpose of this investigation was to determine if computer disruptions affected student scores
  • Specifically, the concern lies in disrupted student scores being lower than expected
  • We investigated multiple groups of interrupted students (grades 6-8 and high school) using multiple methods in an attempt to “converge” on a bias, if any could be detected.
challenges to the investigation
Challenges to the Investigation
  • Students were not interrupted randomly or by design
  • Computer interruptions are not that uncommon, even when there is no identified issue during testing to be discovered
  • Individual students may have responded very differently to the interruption
structure
Structure
  • Four “Cohorts” were investigated
    • Cohort A – All students who had an interruption in the Grades 6-8 dataset, regardless of day
    • Cohort B – Only examined those with interruptions on 4/29 and 4/30
    • Cohort C – EOI Data for Algebra scores
    • Cohort D – EOI Data for English scores
propensity score matching
Propensity Score Matching
  • Propensity scores used to “mimic” an equivalent sample for comparison to the interrupted group
  • Matched the interrupted sample with individuals in the non-interrupted group who were similar on all available variables that relate to 2013 scores, including:
    • Prior year scores
    • School-level scores
    • Ethnicity
    • Gender
    • Free/Reduced Lunch
algebra and english ii eoi
Algebra and English II EOI
  • Two sets of analyses were conducted for each EOI exam
  • Algebra
    • Grade 9 students in 2013 with Algebra scores matched to their Grade 8 Math scores from 2012
    • Grade 7 and 8 students in 2013 with Algebra scores matched to their 2013 math (and reading) scores
  • English II
    • Grade 10 students in 2013 with English II scores matched to their 2013 Grade 10 US History scores
    • Grade 10 students in 2013 with English II scores matched to their 2011 Grade 8 reading (and math) scores
analyses
Analyses
  • All analyses performed on the matched Disrupted and Non-Disrupted groups
  • Mean differences on 2013 scores (Statistical, Meaningful)
  • R2 differences when predicting 2013 scores separately
  • R2 change when combining groups and adding dichotomous disruption variable
  • Applying Non-Disrupted regression equation from step 2 on Disrupted group as well as 5th, 10th, 90th, and 95th percentile cuts
conclusion
Conclusion
  • The evidence shows that the effect of disruption on students’ scores was neither widespread nor large and the conclusions were not consistent across methods.
  • In addition, the results show that the effects did not always disadvantage the disrupted students, but at times the disrupted students did better than expected.
  • The most significant disruption impact seems to be for Algebra students in Grades 7 and 8, but the impact was relatively small and inconsistent across the distribution of students. Lower performing students tended not to perform as well as predicted.