MCAS Reporting: Highlights, Transition, and Tips

1. 2019 MCAS Reporting Call for Principals and Test Coordinators September 20, 2019Embargoed until 12:01 a.m., September 24

2. Logistics for This Session • Questions may be asked at any time using the Questions feature on the GoToWebinar screen. • Questions about a specific student should be sent by email to mcas@doe.mass.edu. • Some questions may be covered during the course of the presentation. • We will answer some questions at the end of this session. • After the session, we will email questions and answers to participants. • This session is being recorded and will be available online in about one week at the MCAS Resource Center.

3. Presenters • Michol Stapel, Associate Commissioner for Student Assessment • Bob Lee, MCAS Chief Analyst • Scott Kelley, Data and Reporting Specialist • David Ragsdale, Test Security Specialist

4. Today’s Call • MCAS reporting highlights • Statewide results for all grades and subjects • Transition to next-generation grade 10 tests • Release schedule and additional resources

5. 2019 MCAS Reporting Highlights • Overall gains across grades 3–8 between 2017 and 2019 • Grade 3 ELA: Meeting Expectations rose from 52% in 2018 to 56% in 2019 • Essay scores rose by almost a point per essay • Grade 6 Mathematics: Meeting Expectations rose from 47% in 2018 to 51% in 2019 • Open response scores up 7% • First math test in grades 3–8 to surpass an average scaled score of 500 and 50% Meeting Expectations

6. 2019 MCAS Reporting Highlights (continued) • Grade 10 ELA and Mathematics • First year of next-generation tests with new scaled scores & achievement levels • Graduation and scholarship eligibility standards and rules remain the same as the legacy standards for at least the classes of 2021 and 2022 • 87% of students passed the MCAS testing requirement in all three subjects on the first attempt (same as last year) • Transition to computer-based testing nearly complete • Only high school STE and retests were paper-based in 2019

7. 2019 MCAS Participation Rates by Subject and Mode

8. English Language Arts Achievement Results 2019 ELA by Grade 2017–2019 Change

9. Mathematics Achievement Results 2019 Mathematics by Grade 2017–2019 Change

10. Science and Technology/Engineering Legacy Grade 10 2019 Next-Generation Gr. 5 & 8

11. Tips for reporting next-generation results at the group level • Use average scaled scores to compare groups • It is still appropriate to use percentages (such as percent Meeting or Exceeding Expectations), but average scaled scores are more precise when comparing groups or change over time. • Results can be averaged across grades 3-8, as shown on slides 7 and 8 • Grade 10 results should not be averaged with lower grades. • For the state release, ESE defined improving as gaining 2.0 or more scaled score points, or more than 2% gains. Statistical significance varies depending on school size. • Student growth percentiles can be averaged across all grades

12. 2017 to 2019 School Level Scaled Score Changes in Grades 3-8 English Language Arts Mathematics

13. Grades 3-8 Results by Race, 2017-2019

14. Changes in Scaled Scores by Race, 2017–2019

15. Grade 3 ELA Results – Essay Scores Results by district District results by poverty rate • 99% of districts saw a reduction in zero-scored essays • The number of students receiving zeroes on one of their two essays dropped from 40,400 (58%) to 16,800 (25%) • Similar gains were seen in high poverty schools and low poverty schools

16. Transition to Next-Generation Grade 10 ELA and Math Tests • First administration of next-generation tests in 2019 • Based on Massachusetts 2017 revised curriculum frameworks • Computer-based • Aligned to NAEP and evidence-based definitions of college & career readiness • Consistent with results established in 2017 for Grades 3-8 • Legacy graduation and scholarship standards remain the same while students become familiar with the new tests • The Department used statistical techniques validated by expert educators to ensure that the knowledge, skills and abilities needed to pass and graduate with a Competency Determination remained similar for at least the next two classes.

17. Legacy Achievement Level Results (2015)

18. Grade 10 Transition: Legacy Compared to Grades 3–8 (2018) ELA Mathematics Next-Generation Legacy Next-Generation Legacy

19. As most Proficient and many Needs Improvement students have attended college… % to College by ELA Grade 10 % to College by Math Grade 10

20. . . . most of the Proficient and very few Needs Improvement students graduated from college within eight years. % Graduating College by ELA % Graduating College by Math Meeting Expectations Meeting Expectations Proficient on Legacy Proficient on Legacy

21. Standard Setting • Needed anytime you introduce a new test • Establish standards for each achievement level; without standard setting, can’t say whether a student met expectations, only whether they answered an individual question right or wrong • Context for standard setting • Legacy MCAS needed to be updated to reflect expectations in revised learning standards, as well as for alignment with grades 3-8 tests • Legacy MCAS high school tests were designed to measure readiness to graduate, not readiness for college and career • Approximately 25% of MA public high school graduates who enrolled in public colleges or universities placed into remedial courses for Mathematics, 10% for reading and writing • In Mathematics 90% of students at 220 required remediation and 50% of students at 250

22. Standard Setting for Grades 5 & 8 STE and Grade 10 ELA & Math • August 5-7, 2019 • 75 educators (with an average of 18 years experience) • Four panels • Grade 10 ELA • Grade 10 Mathematics • Grade 5 Science and technology/engineering (STE) • Grade 8 STE

23. Key Standard Setting Activities • Massachusetts educators worked in grade- and subject-specific groups, lead by facilitators, to make judgments that produced cut score recommendations by identifying the knowledge, skills, and abilities that students need to demonstrate to reach each achievement level • (Grade 10 only) Educators examined student results on the legacy and next-generation tests to verify the interim standard

24. Next-Generation MCAS Achievement Levels • Exceeding ExpectationsA student who performed at this level exceeded grade-level expectations by demonstrating mastery of the subject matter. • Meeting ExpectationsA student who performed at this level met grade-level expectations and is academically on track to succeed in the current grade in this subject. • Partially Meeting ExpectationsA student who performed at this level partially met grade-level expectations in this subject. The school, in consultation with the student's parent/guardian, should consider whether the student needs additional academic assistance to succeed in this subject. • Not Meeting ExpectationsA student who performed at this level did not meet grade-level expectations in this subject. The school, in consultation with the student's parent/guardian, should determine the coordinated academic assistance and/or additional instruction the student needs to succeed in this subject.

25. Achievement Levels and Cut Scores • Partially • Meeting Expectations • Not Meeting • Expectations • Meeting Expectations • Exceeding Expectations • Achievement Less More • Exceeding Expectations • Cut • Score • Meeting Expectations • Cut • Score • Partially Meeting Expectations • Cut Score

26. Educator Validation of Equipercentile Linking • For each cut score, educators compared: • Educators verified that the knowledge, skills, and abilities defined by each of these sets of results were equivalent for each cut score. Student results on a set of items from the 2018 legacy testthat is associated with that legacy cut score (located just below, at, or just above the cut score) Student results on a set of items from the 2019 next-generation test that is associated with the statistically defined interim cut score (located just below, at, or just above the cut score) to

27. Linking Model for Locating Cuts (ELA example) 13% Exceeding Expectations 48% Meeting Expectations % of students 31% Partially Meeting Expectations 8% Not Meeting Expectations

28. Statistical Linking Model (ELA) 13% Exceeding Expectations 48% Meeting Expectations % of students 31% Partially Meeting Expectations 8% Not Meeting Expectations

29. Zooming In on the Pass/Fail Line (ELA) Partially Meeting Expectations 472 240 470 EPP required % of students 5% Passing & Not Meeting Expectations 455 220 3% Not Passing & Not Meeting Expectations 440 200

30. Interim CD Standard for the Classes of 2021 and 2022

31. Interim Adams and Koplik Cut Scores for the Classes of 2021 and 2022

32. MCAS Passing Rates Remain Virtually Identical to Previous Years Number and percentage of students earning a full CD after their first grade 10 attempt

33. Available Resources

34. Upcoming Release Dates (Tentative)

35. Questions & Answers (Use the “Questions” feature to ask questions.)

36. Next Steps • Today: Complete the evaluation form. • Responses associated with the name and email address used to log in • Email your input to mcas@doe.mass.edu if you have problems accessing or completing the form. • Later this week/early next week: • Receive an email with the Q&A from this session • Recording will be available

37. THANK YOU The Office of Student Assessment Services 781-338-3625 mcas@doe.mass.edu www.doe.mass.edu/mcas 75 Pleasant Street, Malden, MA 02148