Implementing Interim Assessment Systems: Best Practices from the Galileo Pilot Project

1. Implementing Interim Assessment Systems: Best Practices from the Galileo Pilot Project A Teleconference Presented by staff from ESE, MAGI Services, Fitchburg Public Schools, and Chelsea Public Schools May 13, 2009

2. Overview of Interim Assessment and the Galileo Pilot Project Presented by Life LeGeros, Director of Statewide Math Initiatives, and Carol Lach, Mathematics Assistance Specialist (llegeros@doe.mass.edu and clach@doe.mass.edu) Department of Elementary and Secondary Education (ESE)

3. Purpose of the Galileo Pilot Project �Evaluate the capacity of an instructional data system to support the systematic improvement of teaching and learning.�

4. Galileo Pilot Project Overview Summer 2005 � Through a competitive process, ESE contracted with ATI to pilot Galileo Online ATI selected because of psychometric expertise and responsiveness to clients SY05-06 � Pilot launched in 8 districts, 25 schools Focus on middle school mathematics Pilot is now entirely district-driven Expanded grade levels, subjects, districts No ESE competitive grant funds after SY07-08

5. Galileo System Features Assessment Item Banks Grades K-12 �Benchmark� bank (secure) ATI draws from this to develop customized benchmark assessments matched to each district�s pacing guide Typical in math � 8 standards: 5 MC items each, + 1 OR �Formative� bank (open access) Teachers can use to develop classroom assessments Items can be modified and new items can be added Assessments administered paper-pencil or online Reports accessible online Other features include standards-based report cards, electronic gradebook, lesson planning tools

6. Purposes of Assessment Inform instructional decisions at each level Institutional Program Classroom Encourage students to learn Ownership of learning Specific feedback Progress builds confidence

7. Institutional Level of Assessment Use Decision to be made � Are enough students meeting required standards? Made by � Superintendents and district leadership, school boards, ESE, legislators Information needed � Annual summaries of standards proficiency on accountability tests Example � MCAS

8. Program Level of Assessment Use Decision to be made � On which standards are our students proficient or not proficient? Made by - Teacher teams, teacher leaders, principals, and curriculum leaders Information needed - Periodic, but frequent, evidence aggregated across classrooms revealing standards not mastered Examples � Interim, benchmark, short-cycle, common assessments

9. Classroom Level of Assessment Use Decision to be made - What comes next in learning and teaching? Made by � Teachers, students, sometimes parents Information needed - Continuous evidence of each student�s current location on the scaffolding leading to each standard (not aggregated) Examples � Questioning strategies, non-evaluative descriptive feedback to individual students, peer assessment, whiteboards, quizzes, exit questions

10. A Balanced System Aligns Assessments to Purposes Important and unique decisions are made at each level, different assessments are appropriate for different purposes E.g., MCAS for accountability determinations Not diagnosing individ student needs at learning standard level Research most strongly supports positive impact of classroom level assessments Continuous and instructionally integrated formative assessment See Black, P.J., & Wiliam, D. (1998) Inside the black box: Raising standards through classroom assessment, Phi Delta Kappan, 80(2), 139-148. http://www.pdkintl.org/kappan/kbla9810.htm

11. State Role in Assessment Institutional level Develop and administer MCAS Use results as part of accountability determinations Program level Programmatic support (e.g., Reading First) Infrastructure (e.g., statewide data warehouse) Initiatives (e.g., Galileo Pilot Project) Guidance documents (e.g., Student Support System for Learning Mathematics in draft form) Classroom level Professional development

12. District Role in Assessment Institutional level Respond to accountability findings Programmatic level Create a balanced system that serves all levels of decision-making Classroom level Ensure that assessments are used to support effective instruction

13. Benchmark/Interim Assessment Supports these Functions

14. A Case Study of an Effective Interim Assessment System Presented by Eileen Spinney, Director of STEM, spinneye@fitchburg.k12.ma.us Fitchburg Public Schools

15. Comprehensive Math Initiative A Bit of History� Ongoing Partnerships � Districts � Expert Providers Vision Developed Conducted PD for Prioritizing Standards �Power Standards� identified K-10 �Unwrapping� Standards Completed K-10 Galileo Benchmark Assessment System Deployed

16. Comprehensive Assessment System Our Goal� Design, implement, and evaluate a tiered assessment system that provides each level of stakeholders accurate, timely, and valid data to inform all decision making. The design of this system will also insure that each assessment result will be an accurate predictor of the success at the next assessment level.

17. Comprehensive Assessment System Our First Steps� District Benchmark/Interim Assessments Led to ? Greater Understanding of Student  Achievement - District facilitated Benchmark  debriefing sessions conducted with every  teacher team ? Aligned Interventions � Re-Teach/Enrich  scheduled for every student ? Formative Assessment � Conducted at  classroom level providing evidence of student  proficiency

18. Comprehensive Assessment System System Components� Impact Students Feedback loops established/emphasized - promoting student management of their own learning and real time information for teachers Interventions � Reteach/Enrich block scheduled for all students Progress monitoring & Overall Growth � Pre-test Post test, school/ grade level common formatives

19. Comprehensive Assessment System Teachers Formalized Debriefing Sessions � Conducted after each Benchmark assessment. Included performance analysis and action planning Professional Development OR Scoring Institutes Assessment Literacy Formative Assessment � Uncovering student misconceptions and overgeneralizations

20. Comprehensive Assessment System System Components� Impact cont�d Programmatic Ongoing review and revisions of curriculum and benchmark planners Expanding/customizing common formative assessments To inform additional instruction on specific standards To provide evidence of student success or additional instruction Sustainability Expand to primary grades (K-2) Expand to include ELA, Science Implement standards-based reporting for parents

21. Assessment as a System � Not an Event Building a system emphasizing assessment as a series of interactive opportunities, all of which together build a comprehensive and vibrant image of student learning, provides the evidence necessary to support evaluation while preserving the integrity of the connections among teaching, learning, and assessing � none of which can happen well or responsibly without the others in place Supporting Mathematical Learning, Jossey-Bass Teacher, 2008

22. Galileo Instructional Data System Pilot Project Evaluation: 2006-2008 Presented by Shelly Menendez smenendez@measinc.com MAGI Services, A Measurement Incorporated Company

23. Overview of the Evaluation The evaluation was conducted from 2006 to 2008 (2nd and 3rd year of the pilot). This presentation includes findings from the 2007-08 final year evaluation Data collection methods included: Surveys to math teachers, principals, and district staff from all participating schools Individual student data on benchmark assessments and MCAS Mathematics Teacher survey data were linked to students� benchmark assessment data, which allowed us to perform sophisticated statistical analyses (e.g., HLM)

24. Model for Use of Benchmark Assessment Data

25. Model for Use of Benchmark Assessment Data Participation: Teachers who were more likely to participate in the development and review of benchmark assessments (including identification of target standards and analyses of benchmark data) were also more likely to use benchmark assessment data to inform their instruction Value: Teachers who strongly agreed that the Galileo assessment system addressed an important need in their school were also more likely to use the assessment data to inform their instruction

26. Support for Use

27. Implementation

28. Model for Use of Benchmark Assessment Data

29. Teachers were designated into two groups based on the extent to which they used benchmark assessment data to inform their instruction (as per the teacher survey data). Teachers who scored at the 33rd percentile or lower on benchmark use were designated into the low group whereas teachers who scored in the 66th percentile or higher on benchmark use were designated into the high group. Using HLM analyses, the table shows that students from the high group scored 15 percentile points higher on the 4th benchmark assessments than students from low group, after controlling for students� prior test performance and teachers� experience.

30. Model for Use of Benchmark Assessment Data

31. Regression analyses showed that benchmark assessment scores at each of the four assessment periods were significant predictors of student performance on MCAS Mathematics. The table presents the correlation coefficients for each exam with MCAS. In other words, students who performed well on the benchmark assessments were also likely to perform well on MCAS.

32. Implementation of an Interim Assessment System: Pointers and Priorities Presented by Kathy Foulser, Mathematics Coordinator kfoulser@comcast.net Chelsea Public Schools

33. Strong Leadership and Broad Support District Level Champion Support from District leadership Clear vision Good working relationships with central office, principals, coaches, and teachers Broad-based support Grade level teacher teams Coaches and lead teachers Technology infrastructure PCs and Scanners

34. Benchmark Assessment Decisions Philosophy: Benchmarking v. mini-MCAS Benchmarking: test what you teach Mini-MCAS: Test progress v. Annual goals Who will take tests? Include ELLs? Special Education classes? New students? Will all students at same grade level take the same test? Administration method Paper and pencil � Simple, comfortable, but uses lots of paper On line � computer lab access or one laptop per child; can randomize question order Individual response pads � one pace for all

35. Benchmark Decisions, continued Frequency Quarterly or 3 times per year Entrance/exit tests Composition: Multiple choice, short answer, open response Test length: 35+ for consistency, reliability Number of questions per standard Relative weight of open response v. multiple choice Scores: ease of interpretation

36. Beginning Benchmark Assessments Selecting �power standards� to test Grade level teacher teams select the standards for each testing period Criteria for selection of standards Durability � value beyond a single test Leverage � value in multiple disciplines Readiness for next level of learning Possibility of re-testing standards Technology side: Purchase Scanners & download Scanline Upload student/class/teacher data to Galileo

37. Moving the Benchmarks from Theory to Practice Reviewing the draft exams Initial review: Teachers review/comment on each question Individually -- convenient As a group -- opportunity for professional learning about the breadth of the standards Final review: One administrator reviews the teacher comments and accepts, replaces, or rejects each question, providing feedback to ATI. ATI �finalizes� the exam and publishes online.

38. Preparing for the Benchmark Tests Administrator schedules test for each class Students receive bar-coded answer sheets Plain paper sheets printed/scanned in house Bubble sheets for multiple choice Worksheets for open response Decide who is responsible for printing and scanning Challenges to paper-and-pencil administration Importance of matching student to bar coded sheets Plan for how to handle missing answer sheets Pre-printing of worksheets with grids, number lines Promptness in turning in scan sheets

39. Scoring and Reporting Results Administrator to check for completeness of scanning Quick (2-3 day) turnaround time for preliminary results based on multiple choice responses Report district-wide and school-wide progress to district and school administrators and to teachers Celebrate schools or clusters that have made great progress Provide teachers with rubrics for open responses and rosters for reporting results Scoring institutes on Saturdays after exams Request scores within 1 week of exam Report results on district- and school-wide basis

40. Using the Results for Tracking Student Progress and Improving Instruction Cluster meetings with coaches to review results Print out results for them to get started Gradually move teachers to independent access of reports Repeatedly provide teachers with detailed instructions for accessing reports on-line Email individual teachers or clusters to celebrate their strong results ask them to share what went right

41. Using Results for Programmatic Change

42. Adding Formative Assessments Grade level teams to set assessment calendar Decide length and composition of assessments Core group trained in developing assessments Grade level teams for assessment review Clear and consistent message re: Purpose of formative assessment Testing window � how much flexibility? Use of results Mechanism for adjustment and feedback Content, Calendar

44. Keys to Success Coordinator to make final decisions, shepherd the system Core team in each school or grade level Technical proficiency Responsibility for managing printing, scanning Use of data so purpose is clear Relentless encouragement/facilitation for teachers to move to on-line scoring and report access

45. Thank you! Please send follow-up questions to galileopilot@doe.mass.edu Presentation and FAQ to be posted at http://www.doe.mass.edu/omste/ca.html