Does the Mathematics Recovery Program Improve Students' Mathematics Learning?

Does the Mathematics Recovery Program Improve Students' Mathematics Learning? Thomas Smith, Paul Cobb, Dale Farran, David Cordray and Charles Munter Vanderbilt University The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305B070554 and R305B040110 to Vanderbilt University. The opinions expressed are those of the authors and do not necessarily represent views of the U.S. Department of Education.

Framing the Problem • Children enter school with a wide range of math abilities • Baroody, 1987; Dowker, 1995; Gray, 1997; Griffin & Case, 1999; Housasart, 2001; Wright, 1991, 1994a; Young-Loverage, 1989 • Children who are less ready for school typically come from low SES families, are of racial/ethnic minority backgrounds, have parents who do not speak English • Alexander & Entwistle, 1998; Berends et al, 2005; Crosnoe, 2005 • Pre-K math ability is highly predictive of achievement at the end of first grade and even at the end of 5th grade • Duncan, Claessens, and Engel, 2004; Princiotta, Flanagan, and Germino Hauskens, 2006 • In the absence of intervention, the initial gap in mathematics achievement continues to widen • Aunola, Leskinen, and Lerkkanen, 2004

An Early Mathematics Intervention • Math Recovery (MR) is a early arithmetic intervention that involves diagnostic, one-to-one tutoring. • MR was first implemented in the United States in 1999 and has served over 3,000 students in 19 states. • How does it work? • Tutors are typically selected from among elementary teacher in a school implementing the program (½ time release ) • Tutor Training • Pullout, one-on-one tutoring • Up to 55 half-hour tutoring sessions every day over an 11 week period • Each session is videotaped; tutors review to plan next session • Each tutor works with 9-12 students per year

MR Program Logic Model Tutor training in Learning and Instructional Frameworks Initial diagnostic assessment Student profile assigned using MR Learning Framework Tutors pose tasks within student’s zone of proximal development MR Instructional Framework used to identify appropriate types of instructional tasks Students develop increasingly sophisticated strategies for solving number problems Students catch up to their peers and are successful in regular math classes

What does MR tutoring look like?

Research Questions • Does participation in MR raise the mathematics achievement of low performing first-grade students? • MR claim— “students performing below the 25th percentile will benefit most from Math Recovery” • Are any gains made from participation in MR maintained through the end of second grade? • We also test the extent to which tutor knowledge and fidelity of implementation influence the effectiveness of MR.

Setting 20 elementary schools (five urban, ten suburban and five rural), representing five districts in two states Each school was a ‘fresh site’ Participants Students were selected for participation at the start of first grade based on kindergarten teacher recommendations and their performance on MR’s screening interview and follow-up assessment interview Students with IEPs, speech/language support, ESL support, or excessive absences were excluded The number of students eligible for tutoring ranged from 17 to 36 across each of the the 20 schools. Research Setting and Participants

Multisite randomized field trial • From the pool of students identified as eligible for MR within each school, students were randomly assigned to a tutoring cohort of 3 students each with a different start date • Cohort A—September • Cohort B—December • Cohort C—March • The remaining students assigned to the “waiting list” • selected to join an MR tutoring cohort if an assigned participant left their school or were deemed “ineligible” due to a special education placement • Process repeated in year 2

Tutor selection • District coordinators recruited 18 teachers to receive training and participate as MR tutors from the participating districts—all of whom had at least two years of classroom teaching experience. • 16 of the tutors received half-time teaching releases to serve one school each; 2 of the tutors received full-time teaching releases to serve two schools each. • All tutoring positions were underwritten by their respective school districts.

Assessments Administered • MR1.1 Screening Assessment • Designed to identify where students are in the MR frameworks and assess eligibility for the intervention • Assessment tasks parallel tasks used in MR tutoring • MR proximal (MRP)-- timed assessment of math fluency, designed in consultation with the program developers • Woodcock Johnson III Achievement tests (WJ III) subtests • Fluency (MF)speed of performing simple calculations for 3 minutes • Applied Problems (AP)--oral, math "word problems,” • Quantitative Concepts (QC)--oral questions about mathematical factual information, operations signs, etc

Figure 1: MR Treatment and Assessment Cycle for Each School MR1.1 = Full Math Recovery Assessment (not given by external assessor) MRp = Proximal Math Recovery Assessment (newly developed) WJ full = Full Woodcock Johnson III Achievement Assessment (Applied Problems, Math Fluency, and Quantitative Concepts) WJ MF = Math Fluency subtest of the Woodcock Johnson III Achievement Assessment only

Sample characteristics of entering 1st graders

Data Analysis • 2-level hierarchical linear model (HLM) with students (i) nested within tutors (j) Outcomeij=β0j+β1j(treatment)ij+β2j(pretest)ij +β3j(female)ij +β4j(lep)ij +β3j(frpl)ij +β4j(age)ij +β5j(Site 1)j +β6j(Site 3)j +β6j(Year2)ij + uj+rij Outcomes MR1.1 (Diagnostic Assessment) MR Proximal WJIII Math Fluency WJIII Applied Problems WJIII Quantitative Concepts WJIII Math Reasoning (AP + QC) Independent variables Treatment Pretest (1st Principal Component of MRP, WJMF, WJAP, WJQC in fall of 1st grade) Female Limited English Proficient (LEP) Free or Reduced Price Lunch Status (frlp) Age at pretest (months) Site1; Site2; Site3 (Site 2 suppressed) Year2 (2007=0; 2008=1)

Model for assessing the effectiveness of MR at the end of 1st grade

Interactions tested • No treatment X year effect • Justifies pooling data across years • Implies tutors are not more effective in their second year • No treatment by site effect • No evidence that tutors in some sites were more effective at increasing student learning than in other sites • Treatment X pretest effect for some assessments • Low performing 1st grade entrants appear to benefit more from math recovery than their higher performing counterparts in mathematical reasoning

How much do the effect sizes change if the sample is limited to students performing below the 25th percentile at the start of 1st grade?

Do the effects of participation in MR last through the end of 2nd grade?

WJIII Applied Problems Predicted Values—start of 1st to the end of 2nd grade

Summary • Positive causal effects on proximal and distal outcome measures (WJIII) from participating in MR at the end of first grade • Some evidence MR is more effective for lowest performing students • Positive effects fade by the end of second grade • Some evidence of long term increases in math fluency and applied problems among participants who started 1st grade performing below the 25% percentile • We cannot attribute the lack of sustained gains through the second year solely to poor implementation • Process components: fairly faithful to the MR model • Structural components: less faithful to the MR model

Tutor Fidelity: Exposure/Duration

Tutor Fidelity: Adherence Perfect Fidelity No Fidelity Assessment 1.1 Profile Assigned Assessment 2.1

Tutor Fidelity: Process Perfect Fidelity No Fidelity Positive Infidelity Ongoing Assessment ZPD Nature of Instruction

Includes instructional practices identified in mathematics education research: • Re-voicing student thinking to highlight particular mathematical ideas, to introduce mathematics vocabulary or to position students in relation to each other and their arguments (Franke, Kazemi, & Battey, 2007; O’Connor & Michaels, 1993) • Asking the student to solve a task (s)he has just solved in a different way, so that the student has an opportunity to approach the same problem from a potentially different mathematical perspective or to represent the mathematics in a different way to find methods that enable progress (Carpenter & Lehrer, 1999: NCTM, 2000) • Asking the student to compare alternative strategies and why they work, to encourage the student to be reflective and to provide opportunities to make connections between various strategies and mathematical ideas (Carpenter & Lehrer, 1999; Rittle-Johnson & Star; 2007). “Positive Infidelity” (Cordray & Hulleman, 2009)

What did we learn from studying Math Recovery? • First evaluation of the causal effects of MR • Tutoring meets goals immediately following treatment, but the effects fade • Why? • Tutor knowledge? • Although tutor knowledge improved among those that started low, no evidence that this improved outcomes • Fidelity of implementation? • Tutors were faithful when doing engaging in strategy based activities • They did not do this very much • Highlights the importance of attending to the developer’s Theory of Action in designing an evaluation

“STRUCTURE” • Exposure/duration • Adherence • “PROCESS” • Quality of Delivery • Participant responsiveness • Program differentiation Administered correctly MR intervention components Initial Assessments Profile assigned 12-15 weeks, 25-30 min Ongoing assessment Instruction Instruction targets ZPD Socratic nature of instruction “Positive infidelity”

Tutor Fidelity: Process Perfect Fidelity = expert mean (n=15) No Fidelity Positive Infidelity *Significant (p<.05) differences between tutors and experts Ongoing Assessment* ZPD Nature of Instruction

Model for assessing the Influence of fidelity to exposure/duration of MR model (end of grade 1) Outcomeij=β0j+β1j(pretest)ij+β2j(lowses)ij +β3j(District 1)ij +β4j(District 2)ij +β5j(Year2)ij + + β6j(# of sessions)ij +β7j(avg session time)ij +β8j(avg strategy time)ij + rij β / sd(DV) ***p < .001 **p < .01 *p < .05

Does the Mathematics Recovery Program Improve Students' Mathematics Learning?