Rate of Improvement Version 2.0: Research Based Calculation and Decision Making

1. Rate of Improvement Version 2.0: Research Based Calculation and Decision Making Caitlin S. Flinn, EdS, NCSP Andrew E. McCrea, MS, NCSP Matthew Ferchalk EdS, NCSP ASPP Conference 2010

2. Today’s Objectives • Explain what RoI is, why it is important, and how to compute it. • Establish that Simple Linear Regression should be the standardized procedure for calculating RoI. • Discuss how to use RoI within a problem solving/school improvement model.

3. RoI Definition • Algebraic term: Slope of a line • Vertical change over the horizontal change • Rise over run • m = (y2 - y1) / (x2 - x1) • Describes the steepness of a line (Gall & Gall, 2007)

4. RoI Definition • Finding a student’s RoI = finding the slope of a line • Using two data points on that line • Finding the line itself • Linear regression • Ordinary Least Squares

5. How does Rate of Improvement Fit into the Larger Context?

6. School Improvement/Comprehensive School Reform Response to Intervention Dual Discrepancy: Level & Growth Rate of Improvement

7. School Improvement/Comprehensive School Reform • Grade level content expectations (ELA, math, science, social studies, etc.). • Work toward these expectations through classroom instruction. • Understand impact of instruction through assessment.

8. Assessment • Formative Assessments/High Stakes Tests • Does student have command of content expectation (standard)? • Universal Screening using CBM • Does student have basic skills appropriate for age/grade?

9. Assessment • Q: For students who are not proficient on grade level content standards, do they have the basic reading/writing/math skills necessary? • A: Look at Universal Screening; if above criteria, intervention geared toward content standard, if below criteria, intervention geared toward basic skill.

10. Progress Monitoring • Frequent measurement of knowledge to inform our understanding of the impact of instruction/intervention. • Measures of basic skills (CBM) have demonstrated reliability & validity (see table at www.rti4success.org).

11. Classroom Instruction (Content Expectations) Measure Impact (Test) Proficient! Non Proficient Use Diagnostic Test to Differentiate Content Need? Basic Skill Need? Intervention Progress Monitor Intervention Progress Monitor With CBM If CBM is Appropriate Measure Rate of Improvement

12. So… • Rate of Improvement (RoI) is how we understand student growth (learning). • RoI is reliable and valid (psychometrically speaking) for use with CBM data. • RoI is best used when we have CBM data, most often when dealing with basic skills in reading/writing/math. • RoI can be applied to other data (like behavior) with confidence too! • RoI is not yet tested on typical Tier I formative classroom data.

13. RoI is usually applied to… • Tier One students in the early grades at risk for academic failure (low green kids). • Tier Two & Three Intervention Groups. • Special Education Students (and IEP goals) • Students with Behavior Plans

14. RoI Foundations • Deno, 1985 • Curriculum-based measurement • General outcome measures • Short • Standardized • Repeatable • Sensitive to change

15. RoI Foundations • Fuchs & Fuchs, 1998 • Hallmark components of Response to Intervention • Ongoing formative assessment • Identifying non-responding students • Treatment fidelity of instruction • Dual discrepancy model • One standard deviation from typically performing peers in level and rate

16. RoI Foundations • Ardoin & Christ, 2008 • Slope for benchmarks (3x per year) • More growth from fall to winter than winter to spring • Might be helpful to use RoI for fall to winter • And a separate RoI for winter to spring

17. RoI Foundations • Fuchs, Fuchs, Walz, & Germann, 1993 • Typical weekly growth rates • Needed growth • 1.5 to 2.0 times typical slope to close gap in a reasonable amount of time

18. RoI Foundations • Deno, Fuchs, Marston, & Shin, 2001 • Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability

19. RoI & Statistics • Gall & Gall, 2007 • 10 data points are a minimum requirement for a reliable trendline • How does that affect the frequency of administering progress monitoring probes?

20. Importance of Graphs • Vogel, Dickson, & Lehman, 1990 • Speeches that included visuals, especially in color, improved: • Immediate recall by 8.5% • Delayed recall (3 days) by 10.1%

21. Importance of Graphs • “Seeing is believing.” • Useful for communicating large amounts of information quickly • “A picture is worth a thousand words.” • Transcends language barriers (Karwowski, 2006) • Responsibility for accurate graphical representations of data

22. Reading Oral Reading Fluency Word Use Fluency Reading Comprehension MAZE Retell Fluency Early Literacy Skills Initial Sound Fluency Letter Naming Fluency Letter Sound Fluency Phoneme Segmentation Fluency Nonsense Word Fluency Spelling Written Expression Behavior Math Math Computation Math Facts Early Numeracy Oral Counting Missing Number Number Identification Quantity Discrimination Skills Typically Graphed

23. Importance of RoI • Visual inspection of slope • Multiple interpretations • Instructional services • Need for explicit guidelines

24. Ongoing Research • RoI for instructional decisions is not a perfect process • Research is currently addressing sources of error: • Christ, 2006: standard error of measurement for slope • Ardoin & Christ, 2009: passage difficulty and variability • Jenkin, Graff, & Miglioretti, 2009: frequency of progress monitoring

25. Future Considerations • Questions yet to be empirically answered • What parameters of RoI indicate a lack of RtI? • How does standard error of measurement play into using RoI for instructional decision making? • How does RoI vary between standard protocol interventions? • How does this apply to non-English speaking populations?

26. How is RoI Calculated? Which way is best?

27. Multiple Methods for Calculating Growth Visual Inspection Approaches • “Eye Ball” Approach • Split Middle Approach • Tukey Method Quantitative Approaches • Last point minus First point Approach • Split Middle & Tukey “plus” • Linear Regression Approach

28. The Visual Inspection Approaches

29. Eye Ball Approach

30. Split Middle Approach • Drawing “through the two points obtained from the median data values and the median days when the data are divided into two sections” (Shinn, Good, & Stein, 1989).

31. Split Middle X(14) X (9) X(9)

32. Tukey Method • Divide scores into 3 equal groups • Divide groups with vertical lines • In 1st and 3rd groups, find median data point and median week and mark with an “X” • Draw line between two “Xs” (Fuchs, et. al., 2005. Summer Institue Student progress monitoring for math. http://www.studentprogress.org/library/training.asp)

33. Tukey Method X(14) X(8)

34. The Quantitative Approaches

35. Last minus First • Iris Center: last probe score minus first probe score over last administration period minus first administration period. Y2-Y1/X2-X1= RoI http://iris.peabody.vanderbilt.edu/resources.html

36. Last minus First

37. Split Middle “Plus” X(14) X(9) (14-9)/8=0.63

38. Tukey Method “Plus” X(14) X(8) (14-8)/8=0.75

39. Linear Regression

40. RoI Consistency?

41. RoI Consistency? • If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not. • Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student.

42. RoI Consistency? • Last minus First (Iris Center) and Linear Regression (Shinn, etc.) only quantitative methods discussed in CBM literature. • Study of 37 at risk 2nd graders:

43. Technical Adequacy • Without a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model.

44. So, Which RoI Method is Best?

45. Literature shows that Linear Regression is Best Practice • Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using an Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit. • This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987). Shinn, Gleason, & Tindal, 1989

46. Growth (RoI) Research using Linear Regression • Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35, 128-133. • Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507-524. • Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12, 179-193. • Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48.

47. Growth (RoI) Researchusing Linear Regression • Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151-163. • Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23, 223-233. • Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18, 356-370.

48. So, Why Are There So Many Other RoI Models? • Ease of application • Focus on Yes/No to goal acquisition, not degree of growth • How many of us want to calculate OLS Linear Regression formulas (or even remember how)?

49. Pros and Cons of Each Approach

50. Pros and Cons of Each Approach