Impact of interruptions on test scores in indiana
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Impact of Interruptions on Test Scores in Indiana PowerPoint PPT Presentation


  • 75 Views
  • Uploaded on
  • Presentation posted in: General

Impact of Interruptions on Test Scores in Indiana. Richard Hill June 25, 2014. Two Parts. Initial study prior to presentation to Legislative committee Quick Outline Details in paper: http://www.nciea.org/publication_PDFs/ISTEP%20RH072713.pdf Follow-up study

Download Presentation

Impact of Interruptions on Test Scores in Indiana

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Impact of interruptions on test scores in indiana

Impact of Interruptions on Test Scores in Indiana

Richard Hill

June 25, 2014


Two parts

Two Parts

  • Initial study prior to presentation to Legislative committee

    • Quick Outline

    • Details in paper: http://www.nciea.org/publication_PDFs/ISTEP%20RH072713.pdf

  • Follow-up study

    • http://www.nciea.org/publication_PDFs/GainsMadebyInterruptedStudents_RH092513.pdf


Three confounding factors

Three Confounding Factors

  • New policy for retention of students in Grade 3

  • Transition from paper and pencil to computer administration

  • Interruptions


Changes in mean scores over years ela

Changes in Mean Scores over YearsELA


Changes in mean scores over years mathematics

Changes in Mean Scores over YearsMathematics


Change from paper and pencil to computer administration

Change from Paper and Pencil to Computer Administration

  • 2009 and 2010 – < 10%

  • 2011 – 36%

  • 2012 – 71%

  • 2013 – 95%


Change not even across grades

Change Not Even Across Grades

  • 2012

    • Grade 6 – 66%

    • Grade 7 – 86%

    • Grade 8 – 92%


Additional center analyses

Additional Center Analyses

  • School-by-school improvement at same grade between 2012 and 2013

  • School-by-school gain, following same cohort of students across grades

  • Student-level gain by students matched from 2012 to 2013


Ctb analyses

CTB Analyses

  • Group Analyses

    • Overall statewide averages

    • Interrupted vs. non-interrupted within 2013

    • Scores before interruption vs. scores after interruption

  • Individual Analyses

    • Before vs. after interruption

    • Performance predicted from previous tests


Presentation to legislative committee

Presentation to Legislative Committee

  • Was overall finding of no change a function of two factors?

    • Some students adversely affected by interruption

    • Other students taking advantage of interruption to learn answers from outside sources, then changing answers when testing restarted


Model of concern

Model of Concern


Available data

Available Data

  • CTB could provide A+B and C+D

  • That is, they knew at which item student was interrupted, and they knew, by item, whether student had made a change

  • But they couldn’t tell (easily) exactly the time the change was made


Analysis of table

Analysis of Table

  • Cell B is the event of interest—changes made after interruption to items presented before interruption

  • But C = 0, and A should equal D

  • So B = (A + B) – (C + D)


Example grade 3 mathematics session 1

Example—Grade 3 Mathematics, Session 1


Results for grades 3 5

Results for Grades 3-5


Results for grades 6 8

Results for Grades 6-8


Conclusion

Conclusion

  • Concern raised by legislators was confirmed—data show that students changed answers from wrong to right more often after interruption than before

  • Impact on overall results was negligible—less than 0.2 scaled score points (on tests with standard deviations of 50-75)


Making no changes from wrong to right grades 3 5 math only

% Making No Changes from Wrong to Right (Grades 3-5, Math Only)


Average of change from wrong to right grades 3 5 math only

Average % of Change from Wrong to Right (Grades 3-5, Math Only)


Conclusions from second analysis

Conclusions from Second Analysis

  • Students reported by CTB as interrupted had higher rates of change from wrong to right

  • Again, estimated impact is less than 0.2 scaled score points

  • Students reported as interrupted by locals had lower rates of change from wrong to right


Conclusion from second analysis

Consistent with first

Impact on overall results negligible--

Conclusion from Second Analysis


  • Login