The Use of Trajectory-Modeled Growth as Part of Adequate Yearly Progress: One State's Results

1. The Use of Trajectory-Modeled Growth as Part of Adequate Yearly Progress: One State's Results Christopher I Cobitz, Ph.D. Reporting Section Chief North Carolina Department of Public Instruction AERA April 2007 Chicago

2. “What do you mean no school made it using growth this year?” - Computer Programmer who worked nights and weekends to implement the calculation of NC’s Adequate Yearly Progress (AYP) Trajectory Growth

3. In North Carolina, no school that met an AYP target using growth met AYP. • Four schools met a target using the combination of proficient students and those meeting their trajectory target score

4. AYP trajectory growth comes through for schools. • 235 more Math targets were met using trajectory growth than using status alone • 2,564 were met with status • 2,799 were met using either status or trajectory growth • 290 more Reading targets were met using trajectory growth than using status alone • 4,960 with status • 5,250 with either status or trajectory • For schools with at least one grade in the span of 3-8. No high school targets were helped.

5. First, the media ready results need to be addressed. • The same schools that were close enough to their target to meet using confidence interval or safe harbor are the same schools who had targets met using trajectory growth.

6. In terms of Policy implications • we have worked hard to ensure that we do not incorrectly identify a school as being in need of improvement • these schools are already meeting AYP using the other safe guards

7. In personal terms • This is like stepping on the brakes when you see a state trooper even though you are already under the speed limit.

8. Why not give up on the other safe guards and use trajectory growth? • there is a drawback • 235 more math targets were met using trajectory growth than with status alone • confidence interval added 1007 • On the reading side, the numbers were 290 and 1084

9. Another particular issue is that of High School • 10th grade in North Carolina is the highest grade used for AYP performance targets • no targets were met using trajectory growth. • the model tops out since the growth projection all must cross proficiency at that point. • Confidence interval • 350 math targets • 188 reading targets

10. This is the brutal fact of all growth calculations • Pretest data must be collected prior to being able to predict performance or build a trajectory • growth cannot be as universal a protection from type II error as the other methods are • safe harbor compares performance in year 1 to performance in year 2 • by definition the cohorts being compared are at least 33% different • the cohorts for safe harbor are readily 50% different due to the effects of emigration and immigration beyond grade level promotion.

11. Why not just use growth instead of status? • It is not possible to calculate growth for a student in their first year in North Carolina. • It would systematically wash many students out of the accountability system for the first year. • A solution would be to use a pretest at the beginning of the academic year for these students • more testing burden. • We tried it once for high school courses • we found that growth results at the high school level were not as reliable.

12. There is an additional difficulty • Not possible to use trajectory growth of all students • to develop the trajectory with an end point of proficient • the student must not be proficient in the first place • NC ABCs growth can be calculated for any student with the requisite scores. • These results are publicly available at: ayp.ncpublicschools.org.

13. Doing away with status and using purely growth? • Using the AMO proficiency targets as the target percent making growth • using the appropriate measure (trajectory for non-proficient students or ABCs for proficient students) • 419 targets met in math • decrease of 2,145 • 20 for reading • decrease of 4,940 • This does assume the same AMO targets which would not be the case. • Since the end point is still all students by 2014, then at some point the AMO targets would have to converge.

14. ABCs growth is a less rigorous measure than trajectory growth. • Over nine years of cohorts • a student who is performing between the lowest two achievement levels • at the beginning of 3rd grade • the end of 8th grade will score proficient • 6 years • Trajectory growth as it applies to AYP places the student scoring proficient at the end of 6th grade • 2 full academic years earlier.

15. Although not directly tied to trajectory growth • It indicates that more students are scoring proficient than are making the expected level of progress each year • Often detractors of growth modeling claim that making growth is no substitute for being proficient • Even using the lower standard of state average growth (ABCs standard) more students are scoring proficient than are making the growth expectation.

16. Are there any things states considering growth should be aware of? • The data burden is high. • identify the student’s first year in a tested grade • then identify the first test score • then project out to the appropriate grade level for proficiency • calculate a trajectory for the student • trajectory targets are maintained • the target set for a student with her first test is used regardless of intervening performance • for each non-proficient student full trajectories are calculated and performance is compared against those targets • then determine the status of the student

17. The data systems are now trimodal • Proficient • Not proficient • Proficient with trajectory growth • It cannot be done if only the previous year’s scores are used, the full testing history is necessary.

18. Students Top out • NC can only begin trajectories in 3rd grade • the majority of students top out of the trajectory growth system in 6th grade • more students start tested grades in NC in third grade than any other • the most benefit is gained by those schools without grade 6 or above • grade 10 is the highest tested grade and therefore no benefit can be given to schools with that grade.

19. A final issue is that of standard setting. • set standards separately for each grade • the interplay between NC’s growth system and it proficiency is not as simple as one would hope • Student scale scores are converted to a “change scale” (modified method of calculating a z-scale) • The calculation of growth, uses the properties of large population distributions to span from grade to grade • Since the standards for proficiency are set independent, • we have instances where a student scores in the proficient range in one grade, • even after making the expected level of progress, does not score proficient in the next grade

20. What does all this mean? • Consider the school that is in a highly transient corner of North Carolina. • Using a status model, this school is required to take the student who walks in its door with whatever initial status • 1 keep the student from losing the knowledge and skills the student came with • 2 continue teaching the student to maintain the same pace of learning the student had prior to coming to the school • 3 somehow teach the student as many as 3 years worth of material that they had not learned in their previous school plus the year’s worth of material for the current year • Using trajectory growth • lack of pretest score • the school is not helped in the first year • the following year • the school can be credited for helping the child progress at the rate of almost 2 years per academic year • Using trajectory growth the school can be recognized for the effect it is having while getting a student up to grade level, not just credited with the students who are already there.

21. Another Thought • NC met its Math Multiracial target for 3-8 using trajectory growth. • Many effects impacting LEA improvement status (we use 3-5, 6-8 and HS) • Not a trivial issue for those LEAs