A Leader’s Role in Developing & Enhancing Collaborative Data Practices

A Leader’s Role in Developing & Enhancing Collaborative Data Practices Mickey Garrison Oregon Department of Education DaymondMonteith Klamath Falls City School District This training is supported by a Statewide Longitudinal Data Systems grant from the U.S. Department of Education.

Webinar Focus What is a leader’s role in initiating, developing and enhancing collaborative data practices? • What factors can leaders impact directly or indirectly? • What to look for and how to support and develop • Team effectiveness • Team data practices • Team & teacher instructional response

Collaborative Data Use Involves the “collaborative analysis and interpretation of some form of student assessment data… as a mechanism for teacher learning about students’ learning needs and the use of this new knowledge for a range of instructional considerations and responses.” (Cosner, 2012, p. 28). Cosner, S. (2012). Leading the Ongoing Development of Collaborative Data Practices: Advancing a Schema for Diagnosis and Intervention. Leadership and Policy in Schools, 11, 26-65. doi: 10.1080/15700763.2011.577926

What does this have to do with PD? • Goal for collaborative data use is to “set the stage for new knowledge to emerge as participants [teachers] encounter new ideas or discover that ideas that they have held as ‘truth’ do not hold up under scrutiny and they use this recognition as an opportunity to rethink what they know and what they do.” (Earl & Timperley, 2009, p. 2). Earl, L. , & Timperley, H. (2009). Understanding how evidence and learning conversations work. In L. Earl & H. Timperley (Eds.), Professional learning conversations: Challenges in using evidence for improvement (pp. 1-12). New York, NY: Springer Science.

Leaders can intentionally support collaborative data use as a teacher development experience. How do you move teams from merely meeting over data to learning within collaborative data teams?

How would you classify most of the data team meetings you observe? • Meetings where teachers talk about data and assignment of students to interventions, and more rarely how they will change their core instruction. • Meetings where data are discussed and then blame is attributed to any number of factors, but rarely instruction. • Meetings where teachers mostly discuss workday logistics and other issues, and more rarely reflect on data, interventions, or instruction. • Meetings where teachers regularly confront their prior assumptions about the effectiveness of their teaching as supported by evidence (data), share their prior instructional actions, and seek or offer help (as appropriate) to make modifications to future instruction.

How can leaders support teacher and team development through collaborative data use? 1 Initiate: • Initiate/form collaborative teams • Set aside adequate time for teams to meet regularly • Establish and support a culture/expectation of collaboration

How can leaders support teacher and team development through collaborative data use? 2 • Support & Develop • Focus on the areas that research indicates leaders can impact (directly or indirectly). • Learn how to distinguish between effective team practices and ineffective practices. • Learn which data analysis and interpretation practices are effective and which practices are less effective. • Assess teams’ development and intervene to support or develop teams and teachers further. (Cosner, 2012)

In which area do you spend the majority of your time with collaborative teams? • Helping teams have engaging and productive dialogue. • Facilitating data analysis and interpretation. • Facilitating discussions on modifying instruction based on data interpretation. • Other

What does it mean to assess Teams’ collaborative data practices? Leader “Look fors”

Leader ‘Look fors’ • Look for factors in 3 areas that impact collaborative data practices: • Team effectiveness • Data analysis and interpretation • Instructional response

Team Effectiveness: • Leithwood (1998) described work-group effectiveness as ability of a work group [team] to achieve positive outcomes in terms of adult work, adult behavior and adult attitudes. (Cohen & Bailey, 1997 as cited in Cosner, 2012, p. 31).

Team Effectiveness What can leaders look for to assess team effectiveness? • Assess presence of Structural Elements: • Focused meeting agendas • Structure for inquiry process • Assess presence of Interpersonal Process Elements: • Team norms for collaboration/team member behavior • Team roles: facilitator, recorder, timekeeper, process observer, reporter • Facilitation skills

Team Effectiveness What challenges to effective interpersonal processes do you see most among teams? • Lack of team norms for working together effectively. • Lack of facilitation skills among team members or designated team leader. • Lack of structure or focused agenda for identifying and guiding the teams’ work. • Lack of knowledge or experience with process tools that engage team members’ in effective work.

Team Effectiveness Structuring the Work • Support for ensuring structure and focus of group work: • Available through the DATA Project at https://sites.google.com/site/oregontoolkit/ • Meeting Tracker Form • Meeting Notes Form • Team Initiated Problem Solving and Problem Clarification Steps embedded in this form. • Webinar Series: Connecting Teaching and Learning Through Assessment at http://oregondataproject.org/content/webinar-series-schedule

Team Effectiveness Develop and Support Effective Interpersonal Processes Using Proactive Data Tools Uncover possible root causes& interdependencies: • Critical analysis-Ishikawa fishbone, • Critical Incident, • Force Field Analysis, • Listening/Environmental Scan, • Action Research, • Gap Analysis, • Wagon Wheel or some other form of Triangulation, etc. http://oregondataproject.org/content/strand-2-training-materials Segments 11 – 15 are tool specific videos Prioritize next steps • All on the Wall, Criteria Matrix, Dot or Nominal Voting Evaluate past, current and future actions • Flow chart; Continue-Stop-Start Chart; Stoplight http://oregondataproject.org/content/strand-3-training-materials Segments 2, 9, 10, 11

Team Effectiveness Leaders’ Roles in Team Composition • Team composition impacts team productivity and effectiveness (Conley, Fouske, & Pounder, 2004). • Who is in the group (interpersonal processes & collegial trust) • Expertise within the group including content, pedagogy and data analysis and use skills • Presence of or need for skilled facilitator • Assess this aspect of team effectiveness • frequently observe team work sessions • Occasionally hold all team meetings in same room at same time • Have periodic focus group sessions with team leaders and representative team members

Team Effectiveness Leaders’ Actions That Provide Pressure • Establish expectation to achieve results from collaboration. • Embed accountability for data use into school-wide goals and plans. Leaders’ Actions That Provide Support • Establish school-wide vision for data use. • Provide clear task communication. • Manage the borders between school and district efforts—filter, buffer and align. • Distribute leadership.

Team Data Practices Data Analysis and Interpretation What should leaders look for to support data analysis and interpretation?

Team Data Practices Impacting adult actions and likely to impact student actions and outcomes More likely to impact adult actions May or may not impact adult actions It’s more than just data analysis • Connect to Instructional Implications • Modify Instruction, Adult Behavior , Structures to Improve Effectiveness • Collect & Organize and Report Various Data • Analyze & Interpret for Meaning • Most likely to impact changes in student behavior and/or performance

Team Data Practices What percentage of teams that you work with teams make it through the full process? • Less than 25% • 26% - 50% • 51% - 75% • Greater than 75%

Team Data Practices We assume collaborative data use will lead to improvement because we assume that… • Connections to standards are obvious or at least accessible to all teachers, • Teachers automatically know how to relate assessment analysis with specific content and skills, • Teachers automatically generate instructional knowledge about how students learn by engaging in data analysis.

Team Data Practices However… • Data analysis and interpretation are complex tasks. • Teachers must consider the underlying knowledge and skills & the learning progressions that relate to certain items or tasks in assessment. • Teachers must consider how students may vary in their development and in their learning progressions. • Teachers must consider how instruction can influence students’ progress in learning at increasingly proficient levels.

Team Data Practices What can leaders look for to support teacher development in data analysis and interpretation? Look for and support deeper types of analysis

Team Data Practices What analysis/interpretation strategies should leaders see? • Standard or skill analysis (item, question or task analysis) of aggregate & disaggregated results for determining strengths and weak areas • Look at lower-scoring items/tasks to uncover students’ thinking, conceptual understanding or misunderstanding/misconceptions. • Root cause analysis—additional work samples, teacher made tasks/questions designed to draw out possible multiple interpretations.

Team Data Practices Do teams engage in effective analysis? • Change over time • Do teams look at student and group progress over time to understand changes in conceptual thinking or changes in presence or frequency of mistakes or misconceptions? • Appropriate comparisons • Do teams compare group and student performance against meaningful benchmarks such as developmental norms to understand the degree of students’ needs or progress? • Do teams compare similar groups of students to each other to benchmark progress and assess effectiveness of teachers’/teams’ actions? (Boudett et al., 2005; Datnow et al., 2007; Langer & Colton, 2005)

Team Data Practices Or, do teams engage in less effective analysis? • Focus on surface knowledge/skills • Do teams emphasize procedural aspects of knowledge/skills rather than conceptual understanding? • For example do they use item analysis to isolate steps in problem that may have been missed by students without looking for the conceptual issues or how this information informs the learning progression? (Nabors Olah et al., 2010)

Procedural versus conceptual response Rather than looking to uncover if the conceptual understanding of equivalent fractions is present. Adding or subtracting fractions: Focusing on the procedure to convert one of the fractions to a common denominator.

Team Data Practices How would you characterize the effectiveness of the teams you work with in terms of their data analysis? • More ineffective than effective • As ineffective as effective • More effective than ineffective Let’s talk about why you characterized your teams in A, B or C.

Team & Teacher Instructional Response Connecting Interpretation to Instructional Responses What should leaders look for to encourage teachers to connect interpretation of data to their instruction?

Team & Teacher Instructional Response The team conversation is critical. Leaders should look for • Teachers to reflect on and critically analyze the effectiveness of their previous practices based on the evidence (data). • More likely to respond to data rather than personal preferences. • Teachers to use data/evidence to reflect on challenges, successes and resources they brought to a specific instructional task or situation.

Team & Teacher Instructional Response Stoplight Activity What did the data analysis tell us about students’ learning in geometry? Start with reflecting on the instructional tasks that were assessed. Was the geometry instruction effective for students? Did students demonstrate the ability to analyze and compare two-dimensional shapes to distinguish triangles from other shapes? What lessons and activities did I provide for my students to learn how to distinguish triangles from other shapes? Was I explicit in my teaching? Did my activities engage students in discriminating among shapes and comparing the properties of triangles to other shapes, particularly those that were similar, but had at least one defining characteristic that was different?

Team & Teacher Instructional Response Stoplight Activity Part 2 Developing the Instructional Response Team dialogue could be structured using this approach. Hint: it helps to capture this on big paper or on projected computer. Red-Which aspects of instruction didn’t appear to be as effective as planned? Don’t repeat unless seriously modified. Yellow-Caution-what did I miss in terms of student learning progression, instructional approach, etc.? Proceed after team has helped troubleshoot and develop next steps. Green-This appeared to be effective. Continue this strategy where needed or plan next steps to move on.

Team & Teacher Instructional Response From “what” to “how” Look for teachers to demonstrate a developmental progression as they consider how to respond instructionally to analysis. (Cosner, 2011a; Goertz et al., 2009; Means et al., 2010).

Team & Teacher Instructional Response Setting the stage for more mature responses to data • Make the work of teaching public • Norm—teams can be problem-hard but people-soft [critique practice not people] • Provide means for reflection (group and individual) • Develop an inquiry mindset where teachers hypothesize potential causal factors contributing to student performance or behavior • Encourage paraphrasing, clarifying, probing, focusing, reframing questions • Use discussion protocols • Relevant artifacts/data • Guiding questions for considering the artifacts • Structured roles and time • Where needed, seek external support for teams in terms of content and pedagogy.

In Summary • Leaders impact collaborative data use • Leaders can look for, support and develop teams • Team effectiveness • Team data practices • Team & teacher instructional response

A Leader’s Role in Developing & Enhancing Collaborative Data Practices