The State of Material on 2011 Science MSP

1. The State of Material on 2011 Science MSP OSPI Science Assessment Update WERA, December 2010

2. Science Assessment Office • Elaine Wood—Secondary Specialist • elaine.wood@k12.wa.us • Kara Monroe—Elementary Specialist • kara.monroe@k12.wa.us • Troi Williams—Administrative Assistant • troi.williams@k12.wa.us

3. Agenda • 2011 Measurements of Student Progress • Standard Setting • Resources available • Lessons learned from scoring

4. 2011 Measurements of Student Progress Grades 5 and 8 First assessment of the K-12 Science Learning Standards (adopted June 2009)

5. Big Ideas in 2009 Science Standards

6. Science Assessment Development Cycle Develop Test and Item Specifications Scenario & Item Writing Committee Content Review & Bias/Sensitivity Committees Scenario & Items reviewed for content by OSPI/contractor Scenario & Item Pilot Testing Pilot Range Finding Committee Items available for Operational Testing Data Review Committee Pilot Scoring Operational Range Finding Committee Operational Scoring

7. Scenarios and Items • Scenarios provide context for sets of items. • Systems, Inquiry, and Application • Physical, Earth/Space, and Life Science • The items associated with a scenario cover a range of standards & range of cognitive complexity. • Items not connected to a scenario are called “Stand Alone” items, which is clearly marked for students.

8. From WASL to Measurements of Student Progress Embedded pilot items makes the total number of items 35.

9. From WASL to Measurements of Student Progress Embedded pilot items makes the total number of items 40.

10. Test Specifications Grade 5 & 8 The test map designates the following proportions:  • At least 20% Systems (with connected EALR 4 content) • 30% Inquiry • 20% Application (which will apply EALR 4 content) • At most 30% EALR 4 domain knowledge that is not associated with a cross-cutting concepts or abilities* *EALR 4 content may be assessed in questions that only assess EALR 4 or in questions that cut across both EALR 4 and a cross-cutting concept or ability (Systems, Inquiry, Application).

11. More on Science MSP

12. Standard Setting: Summer 2011 Following the process used by Mathematics in grades 3-8 last year.

13. Science Standard Setting Plan

14. Science Standard Setting Plan

15. Science Standard Setting Plan

16. “Contrasting Groups” method affords broader involvement in standard setting Everyone in the state who teaches science in grades 5 and 8 is invited to help set the performance standards. • Performance Level Descriptors (PLDs) are developed for “Basic” and “Proficient” • Web-based training on PLDs is provided to participating teachers • In April, participating teachers rate each of their students as being “Basic or below” or “At or above Proficient” • Two distributions of scores are produced: Scores obtained by students rated as “Basic”, and scores for “Proficient” • Where the two distributions merge represents the separation point between “Basic” and “Proficient.”

17. Students rated as “Basic or below” using criteria in PLD for Basic Number of students Raw score points

18. Students rated as “Proficient or above” using criteria in PLD for Proficient Number of students Raw score points

19. Intersection indicates a region for where “Basic” separates from “Proficient” Number of students Raw score points

20. PLD training • Available via Moodle, February 1, 2011 • Learn to use evidence to evaluate 5th or 8th grade students’ proficiency (other grade levels can participate) • Using Moodle website • Short videos of presentations • All documents available for download • Can accomplish all at once, or in multiple sittings • Free clock hours (3) • Encourage groups of teachers to participate together • To sign up, go to link and create an account • http://moodle.ospi.k12.wa.us/course/view.php?id=19

21. Contrasting Groups Study • Available via EDS, April 11-29, 2011 • After completing the PLD Training, 5th or 8th grade teachers can predict their students’ performance on the Science MSP • 5th and 8th grade science teachers can gain access to the EDS application by contacting their District Assessment Coordinator

22. Resources Available On the OSPI website: www.k12.wa.us/Science/

23. www.k12.wa.us OSPI Science Assessment Webpage www.k12.wa.us/science www.k12.wa.us/Science/Assessments.aspx Select Science Assessment from Assessment Drop-down Select Assessments

24. Science Assessment Resources • Test and Item Specifications—more information about: • Types of scenarios • Test map • Item specifications • Vocabulary lists • Cognitive complexity

25. www.k12.wa.us/Science/Assessments.aspx Test and Item Specs www.k12.wa.us/Science/TestItemSpec.aspx Select Test and Item Specifications Select a Grade

26. www.k12.wa.us/Science/TestItemSpec.aspx Test and Item Specs Pages 5-14 Page 22

27. Science Assessment Resources • Updates for 2011—each grade level contains: • General information on the 2011 test • Sample items for classroom use • Scoring information, including rubrics and sample student responses

28. www.k12.wa.us/Science/Assessments.aspx Updates for 2011 http://www.k12.wa.us/Science/EducatorResources.aspx Select Educator Resources Select a Grade

29. Science Assessment Resources • Online Testing Website: http://wa-online.caltesting.org/ • Demo • Flash player • Tutorial • CAL testing program must be installed • Practice items, same as Update for 2011 documents • Both tools help students become familiar with testing platform, and tests the computer system

30. http://wa-online.caltesting.org CAL online testing Stand-Alone item from 8th grade tutorial Select Downloads or Demos

31. Short answer item from 5th grade tutorial CAL online testing

32. Short answer item from 5th grade tutorial CAL online testing Same item scrolled down to show symbols button and symbols tool pop-up.

33. Science Assessment Resources • Teacher Resource tool—tests based on previous standards: • Types of scenarios • Generic questions • Item specifications (previous standards) • Cognitive complexity • State-wide statistics

34. www.k12.wa.us/Science/Assessments.aspx Teacher Resource Tool www.k12.wa.us/TeacherResourceTool2009-10/default.aspx Select Teacher Resource Tool Select Science Select a Grade and a Year

35. Teacher Resource Tool Type of scenario Generic question, rolls over to item specification Percent correct or mean score

36. Science Assessment Resources • Powerful Classroom Assessments (PCAs)—currently linked to WASL released scenarios and items: • Sample items • Scoring information, including rubrics • New PCAs will be developed from future released items

37. www.k12.wa.us/Science/Assessments.aspx Powerful Classroom Assessments http://www.k12.wa.us/Science/EducatorResources.aspx Select Educator Resources http://www.k12.wa.us/Science/PCAs/default.aspx Select PCA Select grade level

38. Lessons Learned from Scoring

39. Transition to new standards Grade 5 Grade 8 Terms : “system” and “subsystem” “inputs” and “outputs” Human needs leading to inventions Weight as a measurement of the force of gravity Terms “closed” and “open” systems “balanced” and “unbalanced” forces Roles of engineers and scientists in technological advances

40. Common misconceptions (gr. 5) • When the results don’t match the prediction, students struggle with how to report honestly • Students think the mass of an substance changes when the substance breaks or melts • Students think that plants receive food from soil • Students have misconceptions about what makes something a producer in an ecosystem. • Students think genetic characteristics are acquired from the environment

41. Common misconceptions (gr. 8) • Students have trouble comparing a model to the real-world system • Students think the mass of an substance changes when the substance changes state or temperature • Students have trouble calculating speed • Students don’t understand photosynthesis • Students struggle to see the how natural selection leads to changes in a population over time

42. Going well (gr. 5) • Variables questions • 65% to 90% of the time • Students are including data in conclusion items • 28-37% get full points • Between 30% and 40% give each supporting data point • Students are attempting energy transfer and new procedure items • 3.1% to 5.0% blanks

43. Going well (gr. 8) • Variables questions • 66% to 84% of the time • Students are including data in conclusion items • 54% get full points • Between 55% and 58% give each supporting data point • Students are attempting new procedure items • 7.9% - 10% blanks

44. Constructed responses (both) • Students confuse different forces (gravity, push, pull, magnetic) with forms of energy (5th: light, heat, sound, motion, electricity; 8th: light, thermal (heat), chemical, sound, kinetic, electrical) • Students need more help writing their own procedures: • Changing the manipulated variable indicated in the question • Measuring the responding variable indicated in the question • Students need to be more detailed when answering application items; more specific than “it will work” or “it must be broken, so fix it.”

45. Upcoming Events • Performance Level Descriptors training • Via Moodle website (February-March 2011) • Scenario Writing Workshops • 5th and 8th grade (February 2011) • High School (February 2011) • Contrasting Groups Study (April 2011)

46. Upcoming Events • Operational Rangefinding Committees • High School (Olympia, April 2011) • 5th and 8th grade (Cincinnati, Ohio, May 2011) • Pilot Rangefinding Committees (July 2011) • Standard Setting (August 2011)

47. Opportunities for Involvement • PEPPER email list • PreSALTers Enthusiastically Promoting Powerful Educational Resources • Get updates about science happenings • Sign up today with your e-mail address, or e-mail us and ask to be a PEPPER • SALT list • Science Assessment Leadership Team • Anyone can apply • Respond to emails to sit on Item Development committees • Committees are chosen to balance state representation and content expertise

48. Science Assessment Office • Elaine Wood—Secondary Specialist • elaine.wood@k12.wa.us • Kara Monroe—Elementary Specialist • kara.monroe@k12.wa.us • Troi Williams—Administrative Assistant • troi.williams@k12.wa.us