Jinnie Choi Yong Sang Lee Karen Draney University of California at Berkeley

1. Jinnie Choi Yong Sang Lee Karen Draney University of California at Berkeley 2009 AERA Annual Meeting, San Diego April 14, 2009 Principle-based and Process-based Multidimensionality and Rater Effects in Validation of the Carbon Cycle Learning Progression

2. Outline of Presentation • Learning Progression of Carbon Cycle • Complications in Validation • Principle- and Process-based Multidimensionality • Rater Effect • Modeling Multidimensionality and Rater Effect • Understanding the Results • Next Steps

3. Outline of Presentation • Learning Progression of Carbon Cycle • Complications in Validation • Principle- and Process-based Multidimensionality • Rater Effect • Modeling Multidimensionality and Rater Effect • Understanding the Results • Next Steps

4. Learning Progression of Carbon Cycle (1) • The Carbon Cycle Project • A part of the Environmental Literacy Project funded by National Science Foundation • The goal is to integrate Environmental Science Literacy into contemporary K-12 curriculum • Big Idea: Environmental Literacy • Environmentally literate students are expected to be able to apply fundamental principles to processes in coupled human and natural systems • Systems: coupled human and natural systems • Principles for scientific accounts: scale, conservation of matter (both mass and atoms), conservation of energy and energy degradation • Processes: generation, modification, and oxidation of organic carbon

5. Learning Progression of Carbon Cycle (2) • Carbon Cycle Learning Progression Framework • Learning progressions are “descriptions of the successively more sophisticated ways of thinking about a topic that can follow one another as children learn about and investigate a topic over a broad span of time” (Duschl, Schweingruber, & Shouse, 2007) • Lower anchor, intermediate, and upper anchor understandings that define levels • Two principle-based dimensions that define progress variables • Six process-based dimensions

6. Learning Progression of Carbon Cycle (3) • Levels of achievement (performances) Upper Anchor Lower Anchor

7. Learning Progression of Carbon Cycle (3) • Carbon Cycle Learning Progression Framework Upper Anchor (Process Dimension) Tracing Matter / Tracing Energy (Principle Dimension) Lower Anchor

8. Outline of Presentation • Learning Progression of Carbon Cycle • Complications in Validation • Principle- and Process-based Multidimensionality • Rater Effect • Modeling Multidimensionality and Rater Effect • Understanding the Results • Next Steps

9. Complication in Validation (1) • Principle-based Multidimensionality • Tracing Matter and Tracing Energy • Progress variables are described separately and used as scoring rubrics • Process-based Multidimensionality • Photosynthesis, Digestion/growth, Cellular respiration, Decomposition, Combustion, and Cross-process • More like item groups, but crossed with Principle dimension • Rater Effect • Multiple raters rated different combinations of groups of people and sets of items

10. Complication in Validation (2) • Validation of a Learning Progression • Levels of performances • How are the students distributed across levels of performances? • Do the items capture students’ lower level performances as well as higher level performances? • Dimensional structure • Do dimensions exist? Are the dimensions statistically distinguishable? • What are the correlations between students’ performances on different latent dimensions? • How consistently/differently do students perform on different sets of items that measured different latent dimensions? • Is a set of items more difficult or easier when measuring one dimension than others?

11. Outline of Presentation • Learning Progression of Carbon Cycle • Complications in Validation • Principle- and Process-based Multidimensionality • Rater Effect • Modeling Multidimensionality and Rater Effect • Understanding the Results • Next Steps

12. Modeling Multidimensionality and Rater Effects • Multidimensional Random Coefficient Multinomial Logit Model (MRCML; Adams, Wilson & Wang, 1997) • Confirmatory analysis of between-item multidimensionality • Multifaceted Item Response Model (Linacre ,1994) • Examine variation in the harshness or leniency of raters • Examine the fit (or consistency) of individual raters with other raters

13. Outline of Presentation • Learning Progression of Carbon Cycle • Complications in Validation • Principle- and Process-based Multidimensionality • Rater Effect • Modeling Multidimensionality and Rater Effect • Understanding the Results • Next Steps

14. Understanding the Results (1) • Levels of Performances

15. Understanding the Results (2) • Principle-based Multidimensionality • Relative fit test Chi2(37.66, 2) = 5.272e-18

16. Understanding the Results (3) • Principle-based Multidimensionality • Item difficulty estimates

17. Understanding the Results (4) • Principle-based Multidimensionality • Person ability estimates

18. Understanding the Results (5)

19. Understanding the Results (6) • Comparison of rater harshness and weighted fit

20. Outline of Presentation • Learning Progression of Carbon Cycle • Complications in Validation • Principle- and Process-based Multidimensionality • Rater Effect • Modeling Multidimensionality and Rater Effect • Understanding the Results • Next Steps

21. Next Steps • More measurement questions… • How can we model responses if a rater rates differently for items that measure different dimensions? • How can we model responses if a rater rates differently for items that requires different scoring criteria? • How can we model responses when an item is measuring one process dimension but doubly scored based on principle dimensions? • How do we model responses when a set of item is differently perceived by different groups of people? • Informing and revising the four building blocks (Wilson, 2004) … • Construct map • Items design • Outcome space • Measurement models

22. For more information… • Environmental Literacy research groups • Michigan State University • Long Term Ecological Research (LTER) Network • University of California at Berkeley • University of Michigan • Northwestern University • AAAS Project 2061 • Visit our websites at… • Environmental Literacy website • http://edr1.educ.msu.edu/EnvironmentalLit/index.htm • Berkeley Evaluation and Assessment Research (BEAR) Center • http://bearcenter.berkeley.edu Thank You!