California Assessment Literacy Initiative for Improved Student Learning

1. California Assessment Literacy Initiative for Improved Student Learning Margaret Heritage California Educational Research AssociationAnnual Conference November 19, 2009 San Francisco, CA

2. Session Objectives 1- Provide background to the California Assessment Literacy Initiative (CALI) 2- Present goals and framework for CALI 3- Present content ideas and delivery system design ideas 4- Solicit feedback on the above

3. CALI Steering Committee • Gina Koency ( LACOE) • Paula Carroll (SJCOE) • Kathryn Edwards (LACOE) • Karen Greer (LACOE) • Margaret Heritage (CRESST/UCLA) • Mary Tribbey (BCOE)

4. The Question… “Cheshire …,” Alice began rather timidly, “would you tell me please, which way I ought to go from here?” “That all depends a good deal on where you want to get to,” said the Cat Lewis Carroll

5. The Answer… “Cheshire …,” Alice began rather timidly, “would you tell me please, which way I ought to go from here?” “That all depends on what your data tell you,” said the Cat Lewis Carroll

6. “When you have a robust data system and professional development system working hand in hand, there is total alignment for the kids.” Chris Steinhauser, Long Beach Unified School District Superintendent

7. Background

8. Increased Focus on Data Use “The collection, analysis and use of educational data are central to the improvement of student outcomes envisioned by No Child Left Behind (NCLB)” (U.S. Department of Education, 2009, p. vii).

9. Data Use • Data use leads to improvements in student learning (Snipes, Doolittle & Herlihy, 2002; Williams, Kirst, Haertel et al., 2005; Armstrong & Anthes (2001); Cawelti & Protheroe, 2001; LaRocque, 2007; Symonds, 2004; Tognieri & Anderson, 2003).

10. Increased Focus on Data Use • Increased attention paid to data use (Heritage & Yeagley, 2005; Ikemoto & Marsh, 2007; Mandinach & Honey, 2008; Wayman & Stringfield, 2006). BUT • Expansion of data gathering without a corresponding jump in data use (Data Quality Campaign, 2006; Hamilton et al., 2009; Rothman, 2008).

11. Lack of Data/Assessment Literacy • Data literacy “…presumes an accumulating facility with the interpretation of data, not to mention a familiarity with data sources and creativity in assembling relevant data quickly and efficiently” (Knapp, Swinnerton, Copland, & Monpas-Huber, 2006, p. 13). • Administrators and teachers lack skills to use data effectively (Datnow, Park, & Wohlstetter, 2007; Heritage, Lee, Chen, & LaTorre, 2005; Stiggins, 2002).

12. What is Assessment Literacy? (HO) Assessment literate educators: • Design/evaluate assessments • Identify learning targets • Provide differentiated instruction & assessments • Monitor student progress • Organize, analyze, interpret, and use data • Evaluate reliability/validity of assessments • Engage students in assessment process & provide feedback

13. Survey • Administer survey of assessment literacy skills and assessment data use

14. Project Goals (HO) • Build county office capacity • Create and disseminate Web-based assessment literacy professional development modules • Conduct impact studies • Provide guidance to teachers focused on using existing state resources • Provide teachers with clear guidance on how to effectively utilize assessment results

15. Discussion • Do the attributes of assessment literacy seem right to you? • What would you change? • Do the project goals seem right to you? • What would you change?

16. Collaborative Model

17. CALI Professional Development Program: Framework

18. Data Use Processes and Skills Research calls for: • A systematic process facilitating data use (Armstrong & Anthes, 2001; Boudett, City, & Murnane, 2005; Heritage & Chen, 2005) • Strong data-literate leadership at all levels of the system (Mason, 2002; Herman & Gribbons, 2001) • A data-driven culture (Datnow, Park, & Wohlstetter, 2007; Lachat & Smith, 2005)

20. Assessment System • Coherent: built on a well-structured conceptual base • Comprehensive: provides a variety of evidence • Continuous: provides indications of student growth over time (NRC, 2001, p.259)

21. Different levels of granularity for different purposes

22. Annual State Tests Interpretation What students have learned/ have they met the standard? Differences among groups Strengths/weaknesses in Individual’s and groups’ learning Strengths/weaknesses in curriculum/instruction/programs Are our improvement strategies working? Annual State Tests Action Adjustments to curriculum, instruction, programs Monitoring/accountability Reporting Inform professional development needs

23. Quarterly/Monthly Assessments Interpretation Progress monitoring for individuals: have students reached the benchmark? Differences among groups Strengths/weaknesses in individual’s and groups’ learning Strengths/weaknesses in curriculum/instruction Quarterly/Monthly Assessments Action Adjustments to curriculum, instruction Progress monitoring / accountability Reporting

24. Minute-by-minute/Daily/Weekly Action What to do to move learning forward Adjustments to ongoing instruction/learning Feedback Minute-by-minute/Daily/Weekly Interpretation Identify the gap between current status and the desired learning goals Identify individual misconceptions/difficulties Identify missing building blocks

25. Assessments in the System

26. Assessment Knowledge The PD modules will address specific knowledge about the assessments: a) range of assessment task (constructed response, selected response, performance, instructional task) b) purpose of each of the assessments and their potential use c) assessment quality: validity (including alignment), reliability, freedom from bias, and usability d) the need for multiple measures to inform decisions

27. Assessment Use

28. Assessment Skills The PD modules will address specific skills for assessment use: a) establishing learning goals accessible to all students b) interpretation skills, including statistical knowledge c) matching learning opportunities to learners’ needs d) feedback to learners

29. Example

32. Guidelines for Writing Data Statements and Summaries Each statement should: • Communicate a single idea about student achievement • Present the facts objectively rather than state evaluative or explanatory comments • Be short, clear sentences or phrases in everyday language that is easy to understand • Be an independent statement, that is, its meaning should not be dependent on other statements • Represent the data accurately by including relevant numerical data when needed for evidence • Review all of the data statements and identify the most important ideas that convey the story about achievement (Adapted from Van Houten, L., Miyasaka, J., Agullard, K., & Zimmerman, J. (2006). Developing an Effective School Plan: An Activity-Based Guide to Understanding Your School and Improving Student Outcomes. Oakland, CA: WestED)

33. Writing Data Statements and Summaries: Activity • Write a paragraph of statements summarizing the major and important findings. The statements can be in a slightly more narrative style, but still tightly based on data. Important numerical results should be included to support the points made. Avoid including personal judgments and opinions. If you find you are describing why the results occurred, or using the word “because” in your summary, you have moved to interpretation and are no longer summarizing! (Adapted from Van Houten, L., Miyasaka, J., Agullard, K., & Zimmerman, J. (2006). Developing an Effective School Plan: An Activity-Based Guide to Understanding Your School and Improving Student Outcomes. Oakland, CA: WestED)

34. Student Involvement in Assessment • Self-assessment (metacognition and self-regulation) • Peer-assessment • Use of feedback

35. Discussion • What topics should the professional development program include to support educators’ effective use of data?

36. CALI Professional Development Program

37. Online Professional Development • Meta-analyses comparing distance education classes vs. traditional classes highlight the need for deliberate course design for online delivery to be effective, specifically incorporation of interactivity (Bernard et al., 2004; Bernard et al., 2009).

38. Modules: Design Elements • Completed in 3-5 hours • Designed to support collaboration • Include assignments tied to practice • Include pretest, checks for understanding, and posttest • Include videos of related practice activities • Include videos of experts speaking • Links to related resources for additional information • Inclusion of interactive tools that would enable the user to explore the impact of certain decisions

39. Virtual Professional Learning Communities • Professional Learning Communities (PLCs) are: “…structures for continuous learning and use of knowledge in the course of conducting the work of teaching” (Hord and Sommers, 2008, cited in Mundry & Stiles, 2009, p. 9). • “Teachers in every case are learning and working with their peers to situate their learning in real practice” (Mundry & Stiles, 2009, p.9)

40. Electronic Networking and Professional Development • Review of research finds the following benefits to incorporating electronic networking into professional development: 1) Reducing teacher isolation & supporting sharing 2) Encouraging reflection on practice 3) Improving teaching practice 4) Encouraging professional learning communities (Barnett, 2002)

41. Electronic Networking and Professional Development (cont.) • Studies of professional development for secondary science teachers incorporating electronic networking (i.e. blogging, discussion forum, and an online community) found evidence of learning through interactions with others, increased levels of reflective teaching, and increased value placed on learning by participants (Luehmann & Tinelli, 2008; Makinster, Barab, Harwood, & Andersen, 2006)

42. Discussion • What else should we be thinking about with regard to the online delivery system?

43. mheritag@ucla.edu