For a hypothetical “MyGrade3” Personalized Learning for 3 rd Grade Reading

1. Model A-ROI PresentationAn Example from SmarterSchoolSpending.orgA Service of the Government Finance Officers Association For a hypothetical “MyGrade3” Personalized Learning for 3rd Grade Reading

2. Research Questions The Why, What and How of this A-ROI Analysis

3. Why are We Doing This? • What is the problem we were trying to solve? • What is the solution we are evaluating and why? • What question is A-ROI answering? • Who is participating in the analysis? • What time frame did the analysis take place over?

4. What is the Problem We Want to Solve? • State Law requires 3rd graders who are not proficient in reading to go to summer reading camp to catch up. This is expensive and time-consuming. • Our current rate of promotion from 3rd grade to 4th grade is: 85% • Our goal is: 95%

5. What is the Solution we are Evaluating and Why? • Personalized learning provides content to learners in a way that best fits their personal learning style. • MyGrade3 is a large and fast growing program for bringing personalized learning to third graders • Many districts are adopting it • Some early results from MyGrade3 at other districts are promising… • …But the evidence is largely anecdotal • We have no rigorous evidence whether MyGrade3 is or is not cost-effective, so we need to find out!

6. What Question is A-ROI Answering? • We want rigorous evidence that MyGrade3 actually works before investing fully • Does MyGrade3 work? • Are students reaching grade-level proficiency? • Are at least 95% of participants promoted within one year of the intervention? • Is MyGrade3 cost-effective? • Is the cost per additional student promoted reasonable? • How does the cost-effectiveness of MyGrade3 compare to other available choices?

7. Who Participated? • The following 4 of our 8 elementary schools comprised the treatment group • Sweetwater Elementary (SWE) • Whispering Woods Elementary (WWE) • Hidden Valley Elementary (HVE) • Rolling Hills Elementary (RHE) • We matched each elementary with another similar elementary in the district and “flipped a coin” to see who was in the treatment group

8. What was the Timeframe?

9. A-ROI Analysis Results -Implementation Fidelity -Effectiveness in meeting promotion goals -Cost-effectiveness -Summary of highlights

10. Implementation Fidelity • Before we can judge if MyGrade3 works, we must know if it has been implemented correctly • MyGrade3 asks students to complete short activities designed to improve reading abilities • Articles, vocabulary exercises • Research from the MyGrade3 vendor suggests that at least 15 activities per week is needed to optimize proficiency gains

11. Implementation Fidelity 35% of students are meeting the recommended number of activities

12. SWE is a bright spot – 60% fidelity HVE had challenges – only 25% fidelity RHE had 35% fidelity, but also 35% of students missing by a wide margin WWE had many students close, but still only 20% fidelity

13. Student Learning Results • Reminder of research questions: • Are students reaching grade-level proficiency? • Are at least 95% of participants promoted within one year of the intervention? • Reminder of goal: • Current rate of promotion from 3rd to 4th is: 85% • Our goal is: 95%

14. Trends in Promotion Rates Grade 3 to 4 Treatment group had better than average rate (91%) Promotion rates ranged between 85% and 76% since 2009 Control group had average promotion rate (80%)

15. Average Growth in Proficiency Rates All Schools 10 percentage points is “normal growth” Treatment group exceeded normal growth *Combined pre and post test group results

16. SWE had 97% proficiency – the best of our schools and the highest growth HVE hardly had any growth RHE had about average results for the treatment group WWE had about average results for the treatment group

17. Raw Test Scores – Treatment vs Control The average for the treatment is not much higher than the control 150 is “proficient” However, notice that the distribution is much narrower! Kids score consistently better when learning is personalized

18. Financial Results

19. Highlights from the Results • Difference between average post-tests for control and experimental group are unlikely due to chance • 95% chance that this impact did not occur by chance alone • Only 1 in 20 times do these results occur by chance • Meets our standard for of looking for at least 95% certainty • Results for treatment group are an improvement, but we have not met our goal • Some challenges with implementation • MyGrade3 is not an inexpensive option, compared to some of the alternatives.

20. Meet the Students

21. Let’s Meet Jenny • A student at Sweetwater Elementary • Her thoughts on MyGrade3 • “The work is not to easy and not too hard. It is just right!” • “Class is fun – I even sometimes do articles at home” • “I feel better reading in front of my friends”

22. Recommendations

23. Recommendations • Improve implementation fidelity • Learn more about what is happening at SWE. Can we copy and transfer what works? • Focus on improving RHE • Cost-effectiveness of MyGrade3 is marginal • Let’s see if we can improve implementation quality over the next year. • If so, and outcomes improve, bringing MyGrade3 to the other schools might make sense • If not, our money could do more good in other ways