How Teachers Use Data: Organizational and Political Conditions at Schools

1. How Teachers Use Data: Organizational and Political Conditions at Schools Accord Institute for Education Research Ahmet Uludag Ph.D. Ali Korkmaz Ph.D.

2. NCLB • No Child Left Behind (NCLB) (2001) legislation have brought accountability mandate to • states, • districts • and schools.

3. NCLB • More assessments (Stringfieldet al., 2005) • Schedule to the test • Staff to the test • Teach to the test • Motivate to the test • Feed to the test • Students: weary of assessments • Teachers: weary of assessments

4. Teacher Perspective on Data? • What is data for teachers and administrators and how they use the data are important questions to answer to understand data use context and processes. • Number of books checked out of the library each week • Discipline issues per day, week, and month • State test results. • Do teachers view data important enough to incorporate in their instruction? If they do, how? • What support they receive from administrators and district office?

5. What is data? • High stakes testing generates data. • Teachers and administrators sit on multitude of data through: • formative, • summative, • benchmark testing, • homework, • projects • discipline. • There is a dilemma of data richness and richness in data use (Stringfield et al., 2005)

6. The problem? • Educators live in a data rich environment but data is not systematic, consistent, and available in useful format. • Despite magnitude of data, there is a scarcity of rigorous research studies on data-driven instruction (Datnow et al., 2007). • Little (2012) reports that there are “only a number of studies encompassing an analysis of observed data use practice” (p. 150). • The practice of data use is ahead of research (Coburn & Turner, 2012).

7. Research Question? • This study explores : • 1. How teachers use data to align appropriate instructional strategies for student needs? • 2. What expectations teachers have and under what conditions data practice occurs? • 3. What tensions data use results in and how teachers and administrators deal with these tensions?

8. Literature Review • use of data is fast becoming a central tenet of many school- or district-level theories of change (Young and Kim, 2010) • promises and challenges on teacher’s data use (Coburn, 2006) • quantitative prowess (Choppin, 2002), • lack of time (Supovitz and Klein 2003), • data management policies and assessment practices yielding difficult to use data (Choppin, 2002), • school practices misaligned with accountability policies (Ingram, Louis, and Schroeder 2004), • and lack of cohesive organizational goals around data use (Herman and Gribbons, 2001) • Organizational culture and political context of schools greatly affects data use (Datnow et al., 2007; Coburn, Honig, and Stein, 2005; Copland, 2003; Feldman and Tung, 2001; Heritage & Yeagley, 2005; Ikemoto& Marsh, 2007; Marsh et al., 2006; Mason, 2002; Schmoker & Wilson, 1995; Wayman& Stringfield, 2006).

9. Theoretical Framework • organizational routines (Spillane, 2012) • Routines are defined by cultural and political environment • School leadership and district establish these routines • “a repetitive, recognizable pattern of interdependent actions, involving multiple factors” (Feldman and Pentland, 2011, p. 311) • Routines • data meetings, • grade level meetings, • district data meetings, • walk-throughs, • teacher evaluation, • student growth.

10. Theoretical Framework • data use processes (Coburn & Turner, 2012) • noticing, • interpreting, • constructing implications

11. Method • Qualitative Study • Exploratory case study • Purposefully selected sites (Paton, 1990) • One district and 6 public choice schools. • Semi-structured interviews

12. Data • School district (Pine ((pseudonym)) is an urban school district in California • Hispanics are majority at these schools. • ELL and Free-Reduced Lunch percentages are 20% and 60% respectively. • Data collection occurred from December 2011 to June 2012. • Data consists of • 19 interviews with teachers and principals, • Observations of 4 district principal meetings, • Observations of 3 staff meetings at schools focusing on data use. • Semi-structured, role-specific interview protocols guided all interviews.

13. Analysis • We utilized a grounded theory approach (Strauss and Corbin 1994) with an iterative analysis. • We analyzed interview notes through a constant comparative method (Glaser 1965; Lincoln & Guba, 1985). • The findings emerged through the following analysis: • (a) comparison within a single interview • (b) comparison of interviews within the same group • (c) comparison of interviews from different groups (Boeije, 2002). • The unit of analysis was apparent thematic concepts within each interviewee's response.

14. Finding #1 • District Level: • Rolling out • New routines • New processes • Routine: in principal meetings, featuring frequent and varied data reviews. • Routine: Principal sits down with English and Math teachers to go over the data. • Ostensive Aspect (Spillane, 2012): existing in principle • data meetings, • grade level meetings, • district data meetings, • walk-throughs, • Use of test scores in teacher evaluation, • student growth data through MAP • PerformativeAspect (Spillane, 2012): existing in practice • Seating chart based on test scores • Small group tutoring • Peer mentoring • Pull-out students from the electives and assign them to intervention. • After school and Saturday tutoring

15. Finding #2 • Pine case study analysis represents that the district responded to lower academic achievement by proposing student level needs analysis in teacher in-service training prior to school year. • The data reviews continued throughout the school year as it is observed in monthly principal meetings • The grades have little credibility in the face of high stakes testing. • MAP Testing, a computer adaptive test, measuring student academic growth and projecting proficiency in high stakes testing was the major indicator monitored. • The district officials used test results on principals to build pressure while the school administrators exerted the pressure on teachers to • produce better academic results. On the other hand, teacher interviews show that administrators • and teachers creatively design new practices to respond the data.

16. Finding #3 • School Level: • Data discussion between admin team and English and math teachers. • The data analysis was in the agenda of staff meetings and grade level meetings. • Pumped up weekly proficiency reports from teacher with celebration in the staff meeting. • Consultant present data to the staff highlighting the needs and strong points. • Principal utilizes data to pressure the teachers to try different things to engage struggling learners. • Assessment Data is an objective tool for principals. • Principals can put the data in front of the teacher and ask what is going on. • The admin identifies and classifies student needing interventions

17. Finding #4 • STAR Test Analysis Training To Identify Student Needs. • In 2011–2012 with a major drop in API scores, Pine adopted data-driven instruction approach to improve proficiency percentage in STAR test. • The district recruited a consultant to train teachers on how to use state test data setting a routine at in-service trainings prior to school year in fall 2011. • The training offered the CST results graphed by grade level, subject and sub-group asking them to compare their expectations and actual state test results. The teachers identified strong and weak areas per subject and grade level examining graphs. Later, the teachers were given templates to complete to find out students who did not meet the proficiency mark. • The teachers reported the practice very useful while it is a manual process and labor intensive. • Some administrators found it very time consuming. • Other administrators underlined the practice as a culture change at school.

18. Finding #5 • Measuring Student Academic Growth and Projecting Proficiency in the State Test. • The district continued to place data to the core of its operation. • The data from MAP Testing served to track student progress in the academic year at three different times. • The district scheduled MAP testing district wide centrally and monitored MAP test taking for all students. • MAP testing best practices were shared with the staff to motivate students to take the test seriously and to minimize test-proctoring issues.

19. Finding #6 • Data tension: Principal and Teachers • English and Math teachers are more stressful. • Teacher and administrator interviews in low performing schools noted that student engagement motivation is a major issue that schools experience. • The increased testing does not necessarily help teachers to motivate students better. • “It is more of the same thing pointing to the same problem over and over.” “I do not need a test to tell me who is proficient and who is basic. I already know who is proficient or who is not”. • Teachers voiced repeated concerns when presented with lower test results. • “Every time there is new data. The principal is like “Let’s try harder”. We cannot try any harder. We need to try smarter”. • Teachers at high performing schools see data as a confirmation of the right direction that the things are headed. • Teachers at lower performing schools consider repeated data use excessive and threatening especially when the conditions they work in are not conducive to teaching due to challenges such as poverty.

20. Finding #7 • In Class and Outside Class Interventions and Peer Mentoring. • The MAP testing empowers the district and administration to check how many students would be proficient in 2012 STAR test. • The tests allow teachers to drill down on the standards that students do not seem to be mastering. This enables appropriate timely interventions. • Constructive and supportive approach of district and school leadership helps teachers better employ test results directly with little resistance. • Teacher interviews revealed student groups formed based on the growth made. • In some cases, teachers worked with school leadership to design interventions. At Redwood (pseudonym) school within Pine district, students were pulled out from their electives to focus on the core course; mathematics and English subjects. • After school tutoring programs became mandatory for students at and below basic levels. • The school leadership supplemented core classes by building mathematics and English study in computer classes utilizing online resources such as Khan Academy. • At Sequoia school, teachers and school administrators designed a peer mentoring structure. • Students identified as mentors worked with designated student mentees in classes. • The mentee and mentor model stipulated the spirit of competition among students resulting in many mentees to attempt to be mentors.

21. Finding #8 • School-wide collaboration among teachers. • History teachers doing more writing to support English teachers. • PE teacher requiring kids to be proficient to be part of the school basketball team.

22. Discussion • This case study illustrates that data use occur at different levels and it does impact instructional practice in different ways depending on expertise, innovativeness, and collaboration among educators. • The data use practices show that schools establish routines relying on data processes such noticing, interpreting, and constructing implications. Coburn and Turner (2011) suggest that this process needs to result in potential outcomes such as organizational change, change in practice, and student learning. • This study evidences that there are ostensive and performativepractices transforming into organizational changes. • On the other hand, data indicating lower academic achievement complicates data use especially when teacher held accountable.

23. Discussion • Looking and interpreting data is one phase of data use process. District does it with a strong push from the superintendent. • Developing solutions to the problems data suggests and tracking the impact of these solutions are other phases of the process for a complete cycle of data use. • Teacher beliefs, motivation, and knowledge affect data use. Some teachers distrust data sources as inaccurate as other studies suggested (Ikemoto& Marsh, 2007; Kerr et al. 2006) and teacher interview in this study concurs. • Teachers lack expertise in interpreting data (Feldman & Tung, 2001) and time to analyze (Feldman & Tung, 2001; Marsh, 2006). • Teachers need support from administrators who are knowledgeable and committed to data use (Marsh et al., 2006).

24. Discussion • The data from this study indicates that data use is often limited to teacher’s practice. • Teachers and others assign various meanings to data, make inferences from data, create explanations for observed patterns, and imagine useful responses to the patterns they detect. • The challenge is to construct implications and potential outcomes. Classroom level implications and outcomes often require school level policy and practice changes. • Teachers make changes at classroom level on their own already but only systematic school-wide changes leave a greater sustainable impact.

25. Question? • ???