evaluating math recovery investigating tutor learning l.
Skip this Video
Loading SlideShow in 5 Seconds..
Evaluating Math Recovery: Investigating Tutor Learning PowerPoint Presentation
Download Presentation
Evaluating Math Recovery: Investigating Tutor Learning

Loading in 2 Seconds...

play fullscreen
1 / 28

Evaluating Math Recovery: Investigating Tutor Learning - PowerPoint PPT Presentation

  • Uploaded on

Evaluating Math Recovery: Investigating Tutor Learning. Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University. Rationale. As in many interventions, MR relies heavily on the knowledge and practice of the tutors.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Evaluating Math Recovery: Investigating Tutor Learning' - brac

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluating math recovery investigating tutor learning

Evaluating Math Recovery: Investigating Tutor Learning

Sarah Elizabeth Green

Thomas Smith

Laura Neergaard

Vanderbilt University


As in many interventions, MR relies heavily on the knowledge and practice of the tutors.

A substantial part of what “adopting” MR means for districts is the training of tutors.

If tutoring is not effective for students we have to ask the following:

Did tutors learn as intended from their training?

Did tutors conduct tutoring as intended with students?

For the first question, this presentation specifically addresses three key issues:

the effect of prior knowledge on tutor learning during training

the effect of tutor knowledge on the effectiveness of tutoring

change in tutor knowledge over the course of the evaluation study

math recovery mr tutoring
Math Recovery (MR) Tutoring

MR is an unscripted intervention that is adapted to each individual student that is tutored.

Tutors making ongoing diagnoses of students’ current mathematical thinking/strategies design instruction to fall within each student’s zone of proximal development.

Central Tool: MR Learning and Instructional Frameworks

Based on 20+ years of research into the development of children's thinking in the area of early number.

Learning Framework: Used for diagnosis and divided into stages by content.

Instructional Framework: Used to guide tutoring instruction based on a students current stage on the Learning Framework.

tutor knowledge for mr
Tutor Knowledge for MR

Math Content Knowledge

This is a first grade intervention where the main content involves: number words, counting, numerals, structuring numbers, addition, and subtraction .

Math Knowledge for Teaching

Math Knowledge for Teaching (MKT; Hill, Schilling & Ball, 2004) is a specialized form of mathematical knowledge specific to how students are likely to solve math problems, which solution methods mathematically generalize, which will be most beneficial for their learning in the long run, etc.

Knowledge of the MR Program

Specific knowledge the MR Frameworks and Assessments, particularly how to use them to determine students’ current levels and plan tutoring instruction accordingly.

example 12 9
Example: 12-9

Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!”

Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!”

Student 3: “It takes two to get to ten and one more to nine…3!”

Math Knowledge for Teaching:

Understand which mathematical concepts are involved (backward number word sequence, one-to-one correspondence, relationships between quantities)

Anticipate this range of solution strategies and more

Understand the relative sophistication of these strategies

Knowledge of MR:

Naming these strategies, determining a student’s Stage of Early Arithmetic Learning (SEAL) level on the Learning Framework and determining appropriate instruction using the Instructional Framework and other published resources (e.g. Wright, Martland, Stafford, 2006; Wright & Stafford, 2002)

example 12 96
Example: 12-9

Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!”

Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!”

Student 3: “It takes two to get to ten and one more to nine…3!”

Taken from Math Recovery Teacher Handbook, 2007

example 12 97
Example: 12-9

Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!”

Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!”

Student 3: “It takes two to get to ten and one more to nine…3!”

Taken from Math Recovery Teacher Handbook, 2007

description of tutors
Description of Tutors

All fully-certified, white teachers: 17 female, 1 male

17 certified for 1st grade (1 for 7th-12th math)


Mathematics: 3 (two at upper grade levels)

Other Subjects: 2

Special Education: 2

Before becoming MR tutors:

10 classroom teachers, 5 Title I teachers

Other: 1 Special Education, 1 Instructional Coach, 1 Title I Reading

Teaching Experience:

General: Range = 3 to 30 years, Median = 11.5 years

Mathematics: Range = 2 to 30 years, Median = 7.5 years

tutor training study sites
Tutor Training & Study Sites

18 new tutors were trained at two different regional sites.

Training involved a five day summer institute and five additional pull out days in the first months of school.

All “fresh” sites in terms of study schools.

tutor assessments
Tutor Assessments

In order to measure the development of the new tutors the project used two measures at the beginning of the study, at the end of year one and the end of year two:

Mathematics Knowledge for Teaching Assessment1

The MKT was developed at University of Michigan in order to measure Math Knowledge for Teaching.

We used a version of the test specific to elementary number and operations.

MR Tutor Knowledge Assessment (TKA)

The TKA was developed in coordination with MR developers and experts specifically for this study.

It is designed to assess a tutor’s understanding of the MR Frameworks and how to apply them in tutoring and assessment scenarios.

Both are multiple choice scenario based assessments intended to put the test-taker into the context of practice

1 (Hill, Schilling, and Ball, 2004)

descriptive statistics pretest scores
Descriptive Statistics: Pretest Scores

*These initial differences between site are statistically significant.


H1: MR Tutors who begin with a higher level of Math Knowledge for Teaching will more quickly understand how to use the MR frameworks and assessments.

Does MKT pretest score predict TKA pretest score?

H2: Tutors who have a higher level of Math Knowledge for Teaching/greater understanding of the MR frameworks will have a more positive effect on student achievement.

Does MKT pretest score or TKA pretest score predict variation in treatment effect across tutors?

H3: As tutors gain experience tutoring and access to students’ mathematical thinking they will grow in both their Math Knowledge for Teaching and their understanding of the MR frameworks and assessments.

Do tutors’ scores on the MKT and TKA increase over time?

h1 theory mkt and knowledge of mr
H1 Theory: MKT and Knowledge of MR

Because students’ solution strategies are central to the MR Frameworks, we hypothesize that new tutors who know more about ways students typically solve problems in early number or understand the mathematics involved in solving early arithmetic problems (i.e. have more MKT) would be at an advantage in the initial training in terms of experiences to draw on in attempting to understand and use the MR frameworks and assessments.

h1 method results mkt tka
H1 Method & Results: MKT TKA

Linear Regression of MKT pretest scores on TKA scores immediately following training.

Y = 0 + 1 (MKT) + e

R2 = 0.25

h1 controlling for different training
H1 : Controlling for Different Training
  • Site predicts TKA score following training.
  • The R2 for this model = 0.50
  • Site and MKT pretest are significantly correlated (0.57,p=0.01).
  • Trainers reported that the lower starting point of the teachers at site A led to the differences in training.
  • Therefore, some of the usefulness of site as a predictor is likely related to the lower MKT that may have led to the differences in training.
h2 theory knowledge and student outcomes
H2 Theory: Knowledge and Student Outcomes

If being an effective MR tutor requires the types of tutor knowledge that we measured then tutors with higher assessment scores should also be the most effective with students.

This relationship is mediated by what they do with students during tutoring (fidelity).

h2 pretest score effects on students
H2: Pretest Score Effects on Students

End of 1st grade and end of 2nd grade treatment-control comparison using previously mentioned controls (e.g. pretest, race, gender, SES).

Multi-level model nesting students within tutors and testing the significance of tutor pretest scores (on both MKT and TKA) on predicting the variation of the coefficient on treatment.

(Achievement) = 00 + 1(treatment)+2-5 (controls) + 

1 = 10 + 11(Tutor Assessment Score)

Using all outcomes from the evaluation study: WJIII (Math Fluency, Applied Problems, Quantitative Concepts), MR Proximal, and the MR 1.1 Internal Assessment

Given that tutors knowledge may be changing over time we tested this model separately for the end of first grade comparison by year.

h2 mkt pretest on effect of treatment
H2: MKT Pretest on Effect of Treatment
  • In year 2, at the end of first grade the MKT pretest scores predict the treatment effect as measured by the MR 1.1 Internal Assessment.

Treatment Effect Mixed Model

*p< 0.1 , ** p <0.05, ***p < 0.01

  • There were no effects of MKT on treatment effect as measured by external student assessments.
  • There was no effect of MKT on treatment effect at the end of second grade.
h2 tka pretest on effect of treatment
H2: TKA Pretest on Effect of Treatment

TKA pretest score is not a significant predictor of the effect of the intervention on any of the student outcomes.

We know that the relationship between knowledge and outcomes is likely mediated by practice

This points to the importance of future analyses linking knowledge and fidelity for understanding where in the MR model the links are breaking down.

h3 mr and generative tutor learning
H3: MR and Generative Tutor Learning

It is a result of the theory and practice of MR tutoring that leads to the hypothesis of generative tutor learning.

We would not expect other interventions, different in their core practices, to result in tutors learning in and from their tutoring practice in the same way as with MR.

Elements of MR hypothesized to contribute to generative learning:

Focus on understanding students’ thinking (Franke, 1998)

Records and reflection about the link between student strategies and problems posed by tutors

“Space” for tutors to be designers rather than strictly “implementers” (Remillard, 2000)

h3 tutor learning
H3: Tutor Learning

Hierarchical Linear Growth Model with three repeated measures (on both the MKT and TKA) nested within tutors.

The predictor, time, is measured in years and used to predict MKT and TKA. If time is a significant predictor, then teachers improved their scores on these assessments over time, evidence of learning.

There is not formal training after the beginning of year one. Ongoing support was limited in general, but this varied a bit based on site.

Therefore, systematic growth by the tutors over the two years would indicate that they are learning in and from the practice of tutoring.

h3 tutor learning22
H3: Tutor Learning

On both measures there is evidence of tutor learning over time.

Given the prior group differences on these measures, what does including the training site as a predictor on assessment score and as a predictor for rate of learning reveal about differences in learning by training site?

*p< 0.1 , ** p <0.05, ***p < 0.01

2This model also controls for the form of the MKT that the teacher took by using a dichotomy indicating when they took form C (either time 2 or time 3, randomly assigned).

h3 tutor learning on the mkt
H3: Tutor Learning on the MKT

There is no difference in the rate of learning on the MKT by site. However, there is a significant difference in the average MKT score by site (as previously mentioned).

*p< 0.1 , ** p <0.05, ***p < 0.01

h3 tutor learning on the tka
H3: Tutor Learning on the TKA

Possible explanations

Different levels of ongoing support led to different learning rates

Ceiling effect on the TKA assessment (39 points possible)

Teacher learning in practice is curvilinear, where once a certain level of knowledge is reached, either there is little room left to grow OR direct intervention (e.g. more intensive PD) is required to continue to grow

*p< 0.1 , ** p <0.05, ***p < 0.01

summary of findings
Summary of Findings

MKT pretest does predict TKA score after the initial training

This suggests that Math Knowledge for Teaching might be one important consideration in choosing tutors who will learn MR quickly.

MKT pretest also predicts the effect of treatment in year 2 as measured by the MR 1.1

Suggests the importance of Math Knowledge for Teaching for being an effective MR tutor; the details of the conditions need further investigation.

Tutors can continue learning (Math Knowledge for Teaching and knowledge of MR) through the practice of MR tutoring.

This suggests that even when highly skilled tutors may not be available at the beginning of adoption, with experience the tutors can continue to improve in knowledge.

future research broad implications
Future Research & Broad Implications

Future analyses:

Investigating MKT relationship to practice and student outcomes.

Investigating the relationship between knowledge of MR and practice of MR using fidelity of implementation data.

Account for the differences in learning using additional data sources: surveys, teacher lesson plans

For future evaluators of teacher “delivered” interventions:

Those delivering an intervention can learn from more than just their initial training.

This implies that thorough evaluations will have to question static and simplistic views of teachers’ knowledge as something they learn in training and apply in practice.

contact information

Contact Information:

Sarah Green

Graduate Student, Dept. of Teaching and Learning

Vanderbilt University



Franke, M. L., Carpenter, T., Fennema, E., Ansell, E., & Behrend, J. (1998). Understanding teachers' self-sustaining, generative change in the context of professional development. Teaching and Teacher Education, 14(1), 67-80.

Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers' mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11-30.

Remillard, J. T. (2000). Can curriculum materials support teachers' learning? Two fourth-grade teachers' use of a new mathematics text. Elementary School Journal, 100(4), 331-350.

Wright, R. J., Martland, J., Stafford, A. K., & Stanger, G. (2006). Teaching number: Advancing children's skills and strategies: Paul Chapman Educational Publishing.

Wright, R. J., Martland, J., & Stafford, A. K. (2006). Early numeracy: Assessment for teaching and intervention: Paul Chapman Publishing.