Supporting Growth Interpretations Using Through-Course Assessments

1. Supporting Growth Interpretations Using Through-Course Assessments Andrew Ho Harvard Graduate School of Education Innovative Opportunities and Measurement Challenges in Through-Course Summative Assessments National Conference on Student Assessment Monday, June 20, 2011

2. Through-Course Assessments and Growth • Multiple assessments through the school year to support a) instruction and learning and b) inferences about future performance, particularly end-of-year tests and, by extension, career and college readiness. • Paper: Google [Andrew Supporting Growth] http://www.k12center.org/rsc/pdf/TCSA_Symposium_Final_Paper_Ho.pdf • Webinar: Google [Ho Growth Webinar] http://media.all4ed.org/webinar-mar-10 • In this presentation: Two important distinctions for any multiple-timepoint growth application. Then, additional challenges raised by through-course assessments.

3. Some important contrasts in growth use and interpretation Growth Description Growth Projection Gain Scores, Trajectories Status Beyond Prediction Trajectory, Gain-Score Model Regression/Pre-diction Model Where a student was, where a student is, and what has been learned in between. Where a student is, above and beyond where we would have predicted she would be, given past scores. Extend past gains in systematic fashion into the future. Consider whether future performance is adequate Use a regression model to predict future scores from past scores, statistically, empirically.

4. Two Approaches to Growth Description Gain Scores, Trajectories Status Beyond Prediction Status beyond prediction Gain Prediction from previous score (or scores, or scores and demographics) Adding two different students with equal statuses beyond predictions. Adding two students with equal gains

5. Two Accountability Models Therefrom Three students with equal statuses beyond predictions. Three students with equal gains • An “Equal Gain” Requirement: Headline: • New Growth Scores Set Unrealistic Expectations on Highest Scoring Students. • Low Standards for Low-Scoring Students. • An “Equal Status-Beyond-Prediction” Requirement • New “Growth” Scores Expect Less Learning from Highest Scoring Students. • Scores depend on arbitrary decisions

6. Pros and Cons Three students with equal statuses beyond predictions. Three students with equal gains • Pros: Straightforward. Aligns with user intuition about growth. Describes growth and progress along an (ideally) meaningful, relevant scale. • Cons: Defensible vertical scales are difficult and costly to support, can be poorly aligned with statistical predictions. • Pros: Incorporates statistical predictions. Does not require vertical scales. • Cons: Poorly aligned with user intuition about growth. Variables supporting predictions can be atheoretical; variable inclusion/exclusion changes predictions.

7. Two Approaches to Growth Projection Trajectory, Gain-Score Model Regression/Prediction Model • Extends gains over time in straightforward fashion. • With more prior years, a best-fit line or curve can be extended similarly. • Extended trajectories do not have to be linear. • Estimates a prediction equation for the “future” score. • Because current students have unknown future scores, estimate the prediction equation from a previous cohort that does have their “future” year’s score. • Input current cohort data into past prediction equation.

8. Stark Contrasts in Projections Three students with equal projections from a trajectory model The same three students’ predictions with a regression model. Three students with equal projections from a regression model.

9. Stark Contrasts in Incentives Three students with equal projections from a trajectory model Three students with equal projections from a regression model. • Lower initial scores can inflate trajectories: • New Model Rewards Low Scores, Encourages “Fail-First Strategy” • Very intuitive, requires vertical scales, less accurate in terms of future classifications. • Low scorers require huge gains. High scorers can fall comfortably. • New Model Labels High and Low Achievers Early, Permanently. • Counterintuitive, does not require vertical scales, more accurate classifications.

10. Growth and Through-Course Assessments • For low-stakes, through-course assessments, either incorporate vertical scales with gain-score growth information or don’t overpromise on the usefulness of through-course data to describe growth through a curricular domain. • As stakes increase, anticipate and communicate incentives, and adjust weights to discourage both “fail-first” and “inertial status” gaming behaviors, ideally with low but not negative weights on earlier tests. • Do not forget that “career and college readiness” does not solve the problem of setting a meaningful and defensible cut score, nor does it predetermine the type of growth model that projects to that target. • Paper: Google [Andrew Supporting Growth] http://www.k12center.org/rsc/pdf/TCSA_Symposium_Final_Paper_Ho.pdf • Webinar: Google [Ho Growth Webinar] http://media.all4ed.org/webinar-mar-10