Washington State LASER 2008 Evaluation Report

1. Washington State LASER2008 Evaluation Report Dave WeaverKelly Jarvis RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201 dweaver@rmccorp.com (800) 788–1887

2. Presentation Contents • Analysis of 2007 Science WASL Results • School-Level Analysis • Student-Level Analysis • Longitudinal Analysis • Module-Level Assessment Results • Sentinel Site Visits Preliminary Results • Recommendation • Upcoming Activities

3. Analysis of 2007 ScienceWASL Results • Evaluation Question • To what extent did teacher professional development on inquiry-based science instruction contribute to improved student achievement on the Grade 5 and 8 Washington Assessment of Student Learning of science (science WASL)?

4. School-Level Analysis • Data Sources • School-Level Science WASL Data for 2007 • School-Level Mathematics WASL Data for 2006 • School-Level Demographic Data • School-Level Professional Development (PD) Index • Total PD Hours of Teaching Staff for 2-Years Per FTE • Methods: Linear Regression Analysis • Dependent Variable: • Percent of Students Who Met Science Standard on 2007 WASL • Independent Variable: • School-Level 2-Year PD Index • Control Variables: • Percent of Students Eligible for Free or Reduced Price Lunch (FRL) • Students Per Classroom Teacher

5. Student-Level Analysis • Data Sources • 2007 Student-Level Science WASL Results for Grade 5 & 8 with Student Demographics • 2006 Student-Level Mathematics WASL Results for Grades 4 & 7 • Teacher-Level 3-Year PD Index • Total PD Hours Teacher for 3-Years Prior to WASL • Methods: Linear Regression Analysis • Dependent Variable • 2007 Science WASL Scale Score • Independent Variable • Teacher-Level 3-Year PD Index • Control Variables • FRL • Previous Grade Mathematics WASL Scale Score

6. Student-Level Data Map

7. Longitudinal Analysis • Data Sources • 2007 Student-Level Science WASL Results for Grade 8 with Student Demographics • 2004 Student-Level Science WASL Results for Grades 5 • School-Level 3-Year PD Index • Total PD Hours Teacher for 3-Years Prior to WASL • Methods: Linear Regression Analysis • Dependent Variable • 2007 Science WASL Scale Score • Independent Variable • School-Level 3-Year PD Index • Control Variables • FRL • Previous Grade Mathematics WASL Scale Score

8. Longitudinal Data Map

9. Finding 1 • The number of professional development hours in which a student’s science teacher participated was a small but significant predictor of student performance on the science WASL above and beyond what could be explained by socioeconomics (FRL) and the student’s skill level (previous math WASL). • This finding is consistent with earlier studies.

10. School-Level Analysis Results Means adjusted by previous year math scores, class size, and FRL.

11. Longitudinal Analysis Results Means adjusted by Grade 5 Science Scale Score, FRL, Non-Minority, Special Education, and Gifted.

12. Finding 2 • Grade 5 teachers must have participated in at least 18 hours of professional development before an impact on student achievement was measurable, and the impact of the professional development on student achievement increased as the number of professional development hours increased.

13. Grade 5 Multiple Regression Analysis Across Teacher-Level PD Index Categories NOTE: Dependent Variable is the Grade 5 science WASL scale score. Control variables are the previous year math scale score and FRL Results are weighted by the number of students assessed.* denotes significance at the .05 level.

14. Finding 3 • Grade 5 teachers who had participated in science professional development served: • A larger proportion of students eligible for free or reduced-price lunch; • A greater proportion of Hispanic and Latino students; and • A smaller proportion of White students. • These differences may account for the differences in student achievement.

15. Differences in Demographics and Achievement of PD Participants vs. Non-Participants

16. Module-Level Assessment of Student Content Knowledge • Evaluation Questions • What science content knowledge gains do students make as a result of their use of the inquiry-based instructional modules? • What is the relationship between the science PD of the teacher and the gains in student content knowledge?

17. Module-Level Assessment of Student Content Knowledge • Assessment • Obtained Permission to use Horizon Research Assessment • Selected Horizon Research Instruments • Human Body from FOSS • Levers and Pulleys from FOSS • Rocks and Minerals from STC • Electric Circuits from STC

18. Module-Level Assessment of Student Content Knowledge • Administration Plan • Recruit teachers with a range of PD levels • Data collection for each student • Pre and post student scores • Student demographics • Gift certificate compensation to teachers • Data collected spring 2008 • 53 classes completed both pre and post assessments • 1087 students

19. 56 Teachers Recruited

20. Finding 4 • Students demonstrated significant improvement between the preassessment and the postassessment on all 3 modules: on average, student scale scores increased more than 19 points.

21. Module-Level Student Assessment Results

22. Finding 5 • The relationship between student performance and teacher professional development was inconclusive. • The performance of the students whose teachers participated in fewer than 18 hours of professional development was significantly lower than the performance of students whose teachers participated in 18 or more hours professional development. • Students whose teachers did not participate in professional development outperformed students whose teachers participated in some professional development. • Additional data will be collected during the 2008–2009 school year.

23. Student Assessment Gains as a Function of Professional Development

24. Sentinel Site Selection • Schools With Significant LASER Participation • Identified 36 Schools • 13 Schools Refused Participation • Added 11 New Schools • Visited 34 Schools • Defined 2 Groups of Schools Based on Science WASL Change Between 2006 and 2007 • Demonstrated Significant Positive Gains • Demonstrated Little, No, or Negative Gains

25. Sentinel Site Visits • 10 Site Visitors • Each site visit: 1 ½ to 2 days • Principal Interview • At Least 5 Teacher Survey & Interviews • At Least 3 Classroom Observations • Data Collection Instruments • http://www.rmccorp.com/LASERSiteVisits/ • Conducted 3 web-based training session • Round 1: Spring 2008 • Results Reported As Rubric Scores

26. Finding 6 • The site visitors gave the vast majority of the schools visited very high ratings in several areas indicating that the schools served by the LASER Alliances had: • Successfully established a core sequence of inquiry-based instructional modules across all of the grade levels, and • Had materials support system that delivered modules to teachers on time and ready to use.

27. Finding 7 • There were several significant differences between schools whose students demonstrated increased performance on the science WASL compared to schools whose students did not.

28. Schools with Grade 5 Students • Schools whose Grade 5 students’ science WASL scores increased had significantly greater: • Participation in professional learning communities • Time during the workday for school-based professional development • Evidence of positive parent and community support • Evidence of teachers integrating literacy with science

29. Schools with Grade 8 Students • Schools whose Grade 8 students’ science WASL scores increased had: • Time during the work day for school-based professional development • Greater participation in professional development that involved examining student work • Lessons that more often engaged students in discussion based on scientificevidence • Lessons whose content was more significant, accurate and worthwhile

30. Schools with Grade 8 Students (cont.) • Schools whose Grade 8 students’ science WASL scores increased had: • Lessons that more often built on prior ideas and experiences • Lessons whose content was more closely connected to the lesson activities • Lessons that were effective at fostering student understanding • Lessons that included mechanisms to monitor students’ understanding of science content • Lessons that helped students make sense of scientific concepts • Lessons that routinely provided opportunities for students to reflect on their new understanding

31. Finding 8 • The schools that demonstrated an increase in the percentage of students who met the science standards had teachers with significantly fewer years of education experience than the schools with no gains. • This difference was more pronounced in the schools that served Grade 8 students. • Testing for demographic differences between the schools that improved their science WASL scores and schools that did not could not account for the findings. Factors examined include: • School size • Ethnic/racial composition of the student population • Program membership of the students • Teacher education and experience

32. Conclusion • The infrastructure to support the use of a core curriculum of inquiry-based science instructional modules is in place and is functioning adequately in the schools visited. • Although these conditions are necessary for the implementation of inquiry-based science instruction, they are not sufficient to raise student achievement as measured by the science WASL.

33. Recommendation 1 • Ensure that the professional development on research-based instructional practices is consistent and explicit across all of the LASER Alliances • Help teachers understand the elements of effective science instruction and use the modules as a means of carrying out the element with their students.

34. Elements of Effective Science Instruction • Intrinsic Motivation • Relevant, interest to student, resolution of dissonance • Eliciting Student’s Prior Knowledge • Articulating what they already believe about a concept • Intellectual Engagement • Students do the thinking • Use of Evidence • Discourse: Justifications and explanations based on evidence • Sense-Making • Closure, meta-cognition, reflection Banilower, E., Cohen, K., Pasley, J., & Weiss, I. (2008). Effective science instruction: What does research tell us? Portsmouth, NH: RMC Research Corporation, Center on Instruction.

35. Recommendation 2 • Increase support for school-based professional development that helps teachers: • Assume accountability for student learning that results from the use of the modules, and • Collaboratively implement the elements of effective science instruction. • Ample structure and leadership for success

36. Evaluation Activities for 2008-09 • Sentinel Site Visits (30 to 35) • Student Content Assessments • Fall, Winter, & Spring • 7 Instructional Modules • Analysis of 2008 WASL Data • Online Surveys • School Principal Survey • Alliance Directors • PD Provider Online Surveys (3) Rocks & Minerals Human Body Electric Circuits Motion & Design Levers & Pulleys Environments Variables

37. ASK-IT Schools Show What Science Engaged Schools Can Do ASK IT schools had teachers who: Used formative assessment to determine student understanding of key concepts Met in professional learning communities to inform their teaching based on student data Increased their science teaching time so most teachers completed each unit.

38. Questions ??? Dave Weaver RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201 dweaver@rmccorp.com (800) 788–1887