1 / 26

Early Implementation and Impact Findings

Early Implementation and Impact Findings. MDRC and American Institutes for Research. Presentation for Regional Education Laboratories February 7, 2008. ERO Project Background and Evaluation Design Implementation Findings Impact Findings

bridie
Download Presentation

Early Implementation and Impact Findings

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Early Implementation and Impact Findings MDRC and American Institutes for Research Presentation for Regional Education Laboratories February 7, 2008

  2. ERO Project Background and Evaluation Design Implementation Findings Impact Findings Challenges to mounting the demonstration and evaluation What’s next? Topics 1

  3. Project Background 2

  4. NAEP: 70 percent of students enter HS reading “below proficient.” Literacy tied to academic performance and high school success. Review of “promising practices” summarized in Carnegie report on “Reading Next” suggest strategies worth testing. BUT, limited rigorous evidence about “what works.” ED making substantial investment in adolescent literacy: ERO Striving Readers REL projects Content Center activities Background 3

  5. OESE/OVAE Special SLC Grant Competition 10 school districts and 34 high schools $1.25 million per high school over five years $250,000 per high school to cover costs of implementing supplemental literacy programs in school years 2005-06 and 2006-07 Grants awarded in June 2005 Supplemental Literacy Interventions Reading Apprenticeship Academic Literacy (WestEd) Xtreme Reading (KU/CRL) ERO Project Overview 4

  6. Program design features Encompass key elements of promising adolescent literacy programs outlined in “Reading Next” Target population: ninth grade students with reading skills 2-5 years below grade level (not appropriate for most SPED and ELL students) Full-year supplemental course that replaces a ninth grade elective Class time is about 45 minutes daily or 90 minutes every other day One ERO teacher per school teaching 4 sections designed for 12-15 students per class ERO Project Overview (cont.) 5

  7. Evaluation Design 6

  8. ERO Evaluation Design • School-Level Random Assignment • Schools randomly assigned to Reading Apprenticeship or Xtreme Reading within districts prior to grant award • 17 Reading Apprenticeship sites and 17 Xtreme Reading sites across 10 school districts • Student-Level Random Assignment • 2,916 students in target range (2-5 years behind) were randomly assigned to ERO at the start of ninth grade year (57% in ERO group and 43% in non-ERO group) • Random assignment was completed an average of 6 weeks into the school year (range = 3 - 10 weeks) 7

  9. Measuring Student Outcomes • Data Collection • GRADE assessment and student survey administered at the end of ninth grade year (spring of 2005-06) • 83 percent response rate (84 percent for ERO group and 81 percent for non-ERO group) • No systematic difference in baseline characteristics between respondents in the ERO group and non-ERO group • Outcome Measures • Reading Achievement (GRADE assessment; standard score values) • Reading Comprehension Composite • Vocabulary • Reading Behaviors (student survey) • Frequency of school related reading (prior month occurrences) • Frequency of non-school related reading (prior month occurrences) • Use of reading strategies in core classes (4-point scale) 8

  10. Measuring Implementation • Data collection • Based on site visits and classroom observations in second semester of the school year. • Implementation fidelity was assessed on two dimensions: • Classroom learning environment • Comprehension instruction • Classroom observation ratings were obtained for 6 constructs common to both programs, and 7 program-specific constructs • Ratings are based on a 3-point scale • Implementation fidelity • To what extent were classroom practices and activities aligned with model intent and specifications? • Composite ratings for each of the two dimensions were calculated, and then used to classify sites as: • “well aligned” (average rating = 2 or higher) • “moderately aligned” (average rating = 1.5 – 1.9) • “poorly aligned” (average rating = below 1.5) 9

  11. Measuring Student Exposure to Treatment • Data Collection • Attendance records for ERO classes • Student course schedules • Student Surveys • “Dosage” Measures • Enrollment and attendance in ERO classes • Enrollment in courses that the ERO class replaced • Self-reported participation in literacy support activities, in and outside school 10

  12. Year 1 Implementation Findings 11

  13. Implementation fidelity: alignment with program models 16 of 34 schools were well aligned on both classroom learning environment dimension and comprehension instruction dimension (7 RAAL and 9 XR) 8 of 34 schools were moderately aligned on one dimension and moderately or well aligned on the other (4 RAAL and 4 XR) 10 of 34 schools were poorly aligned on at least one of the two dimensions (6 RAAL and 4 XR) Overall, sites were better aligned in terms of learning environment than comprehension instruction Implementation Findings for Cohort 1 12

  14. Implementation Findings (cont.) • Duration and Dosage for Cohort 1 • ERO programs began an average of six weeks after that start of the school year (ranging from three to ten weeks). • ERO programs operated for 7.7 months on average • 95.5% of the ERO group respondents attended at least one ERO class, and 91.2% were still attending at the end of the year • Attendance in the ERO classes averaged 83 percent per month. • The ERO classes replaced a mix of elective classes that did not appear to include literacy instruction. 13

  15. Participation in Supplementary Literacy Support Activities for Cohort 1 14

  16. Year 1 Impact Findings 15

  17. Impact Findings for Cohort 1Reading Achievement • Impact on reading comprehension • Estimated pooled impact for RAAL and XR was .90 standard score points (effect size = 0.09) and statistically significant • Estimated impacts for each program separately were also .90 standard score points, but were not statistically significant individually • Impact on vocabulary • Estimated impact was not statistically significant 16

  18. Estimated Impact on Reading ComprehensionCohort 1 Estimated impact = 0.9* 90 Growth for ERO group: 4.3 3.4 85 80 75 70 100 National average at spring of 9th grade: 100 95 85.9 ERO group mean at baseline Average standard score Estimated impact Estimated growth for non-ERO group ERO group mean at baseline 17

  19. Estimated Impact for Cohort 1Reading Comprehension 100 95 Estimated Impact Estimated Impact = 0.9 = 0.9 90 Growth for ERO group = 4.8 Growth for ERO group = 3.8 2.9 3.9 85 80 75 Estimated growth for non-ERO group ERO group mean at baseline 70 • National average at spring of 9th grade: 100 86.0 85.7 ERO group mean at baseline ERO group mean at baseline Average standard score Estimated impact Reading Apprenticeship Xtreme Reading 18

  20. Impact Findings for Cohort 1 Reading Behaviors • On average, across the 34 schools, estimated impacts on the three reading behaviors were not statistically significant • Estimated impacts for the specific ERO programs should be interpreted cautiously because differences in impacts across the reading behavior measures are not statistically significant (multiple hypothesis tests). 19

  21. Impact Findings: Student Subgroups • Positive and statistically significant impacts on: • Reading comprehension, for students overage for grade and students from multilingual families • Amount of non-school-related reading, for students from multilingual families • Results for individual subgroups should be interpreted with caution because the differences in impacts across subgroups are not statistically significant (multiple hypothesis tests). 20

  22. There is no systematic interaction between impacts and the implementation fidelity scale or between impacts and program duration. The 15 schools with a stronger start-up produced an estimated impact of 0.17 standard deviations (statistically significant) Stronger start-up = Began their programs within the first six weeks of the school year and implementation fidelity was moderately aligned or well aligned with program models Although the difference in impacts across subgroups of sites is statistically significant, no inference can be made about the causes of this difference. Impacts and Implementation Challenges 21

  23. Impacts of Reading Comprehension Cohort 1, by Start-up Implementation Issues • National average at spring of 9th grade: 100 Impact = 1.8* 22

  24. Timing of student identification and eligibility determination End of 8th grade vs. Start of 9th Grade Method of eligibility determination Project-administered reading assessment vs. State or District assessments Universal screening vs. targeted screening Obtaining parental consent The role of school-based staff Investment in program implementation Maximize professional development and coaching vs. “Replicable” level of teacher training and support Demonstration costs vs. steady state costs Key Challenges to Mounting the Demonstration and Evaluation 23

  25. Analysis of implementation and impacts for second cohort of students Second report due in 2008 Follow-up through Grade 11 for Cohort 1 and Grade 10 for Cohort 2 using schools records to measure progress toward graduation and performance on high stakes tests. Final report due in 2009 What’s Next? 24

  26. IES Website www.ies.ed.gov/ncee Report URL http://ies.ed.gov/ncee/pubs/20084015.asp Link to ERO Report 25

More Related