Designing Learning Progression Assessments that Assess Principles First

1. Designing Learning Progression Assessments that Assess Principles First Kathryn Oleszkowicz, Jennifer Doherty, Andy Anderson Michigan State University

2. Item Design Items should reliably assess student understanding of principle-oriented reasoning APPLIED to real-world contexts. How do we design written items that do that?

3. Factors that could influence performance • Context specific knowledge • Scaffolding we provide • Random error • Principle-based reasoning

4. Tracing Matter TREEGROW (Plants): When a tree grows, it makes new wood in its trunk. How do you think that other parts of the tree help the tree to make wood in its trunk? A. How do the leaves help? B. How do the roots help? C. The tree also uses water, air, sunlight, and soil nutrients as it grows. How does it use those things to grow?

5. Oaktree Tracing Matter OakTree: A mature oak tree can have a mass of 500 kg, or more, even after all the water in the tree is removed. Yet it start from an acorn that weighs only a few grams. Where did this huge increase in mass come from? Which of the following statements is true? a. ALL of the mass came from matter that was originally outside the tree, OR b. SOME of mass came from matter that the tree made as it grew. How much of the dry mass came from the AIR? All or most, some, none How much of the dry mass came from SUNLIGHT? All or most, some, none How much of the dry mass came from WATER? All or most, some, none How much of the dry mass came from SOIL NUTRIENTS? All or most, some, none Explain your choices. How does the oak tree gain mass as it grows?

6. Design • Pairs of items that assess students on a similar principle • Items are on the same form of the test, same students take both • Compare level of student performance from first item in pair to level in second item in pair • Examine students’ responses who receive different levels for items assessing similar principle • Find patterns in those students’ responses

7. Representation of Data

8. OaktreeTreeGrow How do you think that other parts of the tree help the tree to make wood in its trunk? A. How do the leaves help? B. How do the roots help? C. The tree also uses water, air, sunlight, and soil nutrients as it grows. How does It use those things to grow? How does the oak tree gain mass as it grows? 67% n = 210

9. KLG Overall Identifying carbon-transforming processes that affect CO2 Why do you think this graph shows that there is more carbon dioxide in the atmosphere each year? How much of the continuing rise is caused by HUMANS BURNING COAL AND GASOLINE? All or most, Some, NoneBy CHANGES IN PLANT GROWTH? By NUCLEAR POWER PLANTS?By CHANGES IN WIND AND WEATHER?Explain your choices. Why is there more carbon dioxide in the atmosphere each year? Context: Large Scale – CO2 in WORLD

10. Identifying carbon-transforming processes that affect CO2 FLBulbs Fluorescent light bulbs use less energy than incandescent light bulbs. Do you think that using fluorescent light bulbs instead of incandescent light bulbs can contribute to slowing global warming? Yes No Explain your answer. How can using fluorescent bulbs help slow global warming, or why will they not help?

11. KLG Overall FLBulbs How can using fluorescent bulbs help slow global warming, or why will they not help? Explain your choices. Why is there more carbon dioxide in the atmosphere each year? 57 % n = 152

12. Level 4 on KLG Overall • The accumulation of fossil fuels in the atmosphere increases each year, and we continue to burn more gas and coal as our society develops and the population grows. Level 2 on FLBulbs • At this point in time, nothing is going on to help slow down global warming so Idoubt fluorescent lights will help Vocabulary in the stem of FLBulbscauses students to talk about their opinions of global warming, rather than what they know about carbon release.

13. Identifying carbon-transforming processes that affect CO2 New FLBulbs Fluorescent light bulbs use less energy than incandescent light bulbs.Do you think that using fluorescent light bulbs instead of incandescent light bulbs canreduce the amount of carbon dioxide going into our atmosphere? Yes NoExplain your answer. How can using fluorescent bulbs help reduce the amount of carbon dioxide going into our atmosphere, or why will they not help?

14. KLG Overall FLBulbs How can using fluorescent bulbs help reduce the amount of carbon dioxide going into our atmosphere, or why will they not help? Explain your choices. Why is there more carbon dioxide in the atmosphere each year? 41 % n = 411

15. NEWEST FLBulbs Identifying carbon-transforming processes that affect CO2 Do you think that turning on a light bulb causes carbon atoms to go into the atmosphere?

16. Written Item Design Authenticity Reliability

17. Thank you! Acknowledgements: Environmental Literacy Group at MSU Andy Anderson Jennifer Doherty Jenny Dauer Hannah Miller Allison Webster Melissa Janos Elizabeth Tompkins Cara Morrison