Reversing The Widget Effect

1. Reversing The Widget Effect Teacher Union Reform Network I November 9 -10

2. Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

3. National nonprofit, founded by teachers in 1997 Partners with school districts, state education agencies, and charter schools Targets acute teacher quality challenges Delivers a range of customized services and solutions on a fee-for-service basis Approx. 200 employees, most embedded in school district offices; majority are former teachers Past and present clients include: Districts: Baltimore, Chicago, Denver, Memphis, New Orleans, New York, Oakland, Philadelphia, San Antonio, Washington, DC States: Alaska, Louisiana, Texas, Virginia The New Teacher Project (TNTP) helps school districts and states fulfill the promise of public education by ensuring that all students—especially those from high-need communities—get excellent teachers.

4. Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

5. Study Scope In Fall 2008 through Spring 2009, TNTP partnered with 12 school districts in four states to analyze each district’s evaluation, tenure, remediation and dismissal policies and practices as well as each state’s teacher performance management policies. • Represented: • Urban, suburban and rural districts • Collective bargaining and non-collective bargaining states • Peer review and traditional evaluation approaches

6. Data Collection

7. Membership One panel per state Approximately 80 stakeholders total Advisory Panel members include: District leaders and HR staff Local and state union leadership School board members State policy makers Education advocates Foundation representatives Three formal meetings, 2-3 hours long. Advisory Panels TNTP’s research benefited from the extensive input and feedback of Advisory Panels in each of the four states studied. • Role • Share information and perspective on state/district context. • Suggest issues for additional exploration and sources of information. • Comment on study methodology and data collection methods and instruments. • React to preliminary findings and recommendations. • Craft brief responses to the study for publication in unedited form.

8. Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

9. “When it comes to measuring instructional performance, current policies and systems overlook significant differences between teachers. There is little or no differentiation of excellent teaching from good, good from fair, or fair from poor. This is the Widget Effect: a tendency to treat all teachers as roughly interchangeable, even when their teaching is quite variable. Consequently, teachers are not developed as professionals with individual strengths and capabilities, and poor performance is rarely identified or addressed.” The New Teacher Project, 2009 The Widget Effect

10. Teacher Evaluation: Where We Codify The Widget Effect

11. When multiple ratings are available, teachers tend to be assigned the highest ratings and are very rarely assigned poor ratings.

12. In districts that use binary “Satisfactory/Unsatisfactory” rating systems, the “Unsatisfactory” rating is almost never used. Evaluation Ratings for Tenured Teachers in Districts with Binary Rating Systems

13. These data often stand in sharp relief against current levels of student achievement, as this example from Denver illustrates. Frequency of Unsatisfactory Ratings in Denver Public Schools that Did Not Meet AYP In Denver schools that did not make adequate yearly progress (AYP) for three consecutive years (2005-06 to 2007-08), more than 98 percent of tenured teachers received the highest rating—satisfactory.

14. Teachers say districts aren’t doing enough to recognize and retain top performers as measured by their impact on student learning… 59% of teachers report their district is not doing enough to identify, recognize, compensate, promote and retain the most effective teachers as measured by their impact on student learning. “All the good quality teachers leave the district after just a few years. They need more incentive to stay.” “Some sort of recognition or praise would be nice. Those doing a good or great job are never told so.” “If you pay the shining stars the same as the slackers, you will dim the shining stars and reinforce the sloth of the slackers.” “I, and others, work hard because we have a conscience, but I don't think [the district] sees us as any different than the lower performing teachers. Teachers who work hard receive very little praise or notice.” TNTP survey of 7,318 teachers across four sites conducted May 2008 to April 2009

15. …or to ensure that evaluation practices and systems provide them with meaningful feedback. 39% of Denver teachers who had a development area identified on their most recent evaluation “do not know” which performance standard they failed to meet. Source1: TNTP survey of 15,176 teachers across 12 sites conducted May 2008 to April 2009 Source2: TNTP survey of 1,863 Denver Public School teachers conducted November to December 2008

16. And despite the fact teachers and principals agree that poor instruction is pervasive… “Are there tenured/non-probationary teachers in your school who deliver poor instruction?” (Percent responding “Yes”) Percent of All Ratings that Indicated “Unsatisfactory” Performance Source: TNTP survey of 7,318 teachers and 751 administrators across 4 sites conducted February to April 2009; teacher evaluation data provided by each district.

17. …dismissal for poor instructional performance almost never occurs. Frequency of Teacher Dismissals for Performance (Non-Probationary Teachers)

18. Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

19. Research has shown that teachers have a greater impact on student learning than any other school-based factor. Dallas students who start 3rd grade at about the same level of math achievement… …may finish 5th grade math at dramatically different levels depending on the quality of their teachers. After 3 EFFECTIVE Teachers 50 After 3 INEFFECTIVE Teachers Original analysis by the Education Trust. Source: Heather Jordan, Robert Mendro, and Dash Weerasinghe, The Effects of Teachers on Longitudinal Student Achievement, 1997.

20. Therefore, dramatic improvements in student achievement cannot occur without a sustained focus on maximizing teacher effectiveness. 5 Boost effectiveness of all teachers through effective evaluation and targeted professional development Goals for Optimizing Teacher Effectiveness 2 1 Optimize new teacher supply by hiring from preparation programs whose teachers consistently achieve higher student outcomes. Retain and leverage most effective teachers 3 Teacher Effectiveness (e.g., Value Add, Growth, Eval. Rating) 4 5 Improve or exit persistently less effective teachers and replace with more effective teachers. Increase the concentration of effective teachers in high-need schools. Current Performance Potential Performance

21. Evaluation system that differentiates teachers based on student academic growth Education leaders must use clear evaluations of teacher effectiveness to inform decisions at each step of the human capital continuum. TNTP Human Capital Continuum Retention / Dismissal Recruitment Compensation Selection Working Conditions Training / Certification Managing Effectiveness Optimizing Supply Evaluation / Prof. Dev. Hiring / Placement School-Level Human Cap. Mngmnt. On- Boarding Supportive policy context that removes barriers and publicizes results / outcomes data

22. Most of the districts we studied considered teacher effectiveness only when making decisions about remediation and dismissal.

23. Outcomes of an Effective Teacher Evaluation System • Improvement: All teachers should report receiving ongoing feedback that helps them improve, and there should be evidence of that improvement in their observations and in student performance. • Higher Standards: Teachers should see a high standard of instruction being upheld for themselves and others, and they should not report instances of poor performance going unaddressed. • Credibility: Teachers should feel evaluations are accurate and credible. • Differentiation: Each school should know which teachers are currently performing at great, good, fair, and poor levels. • Recognition: Examples of outstanding instruction should be recognized and made available to other teachers to help them improve. • Support: Professional development should be aligned to the evaluation process.

24. Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

25. For many years, peer review has been an innovative approach to teacher evaluation, including in Toledo, a Widget Effect study site. Components of peer review: • Early career mentoring and evaluation as a component of the tenure granting process. • Joint labor-management decision-making about renewal, support for struggling teachers and teacher dismissal, usually through the use of lead or master teachers. • Intervention for poor performance where teachers or principals refer tenured teachers for intervention when necessary. Beliefs guiding peer review: • Teachers have a vested interest in the quality of the workforce and will hold their peers to even higher standards than administrators would. • The best way to ensure quality is to provide intensive support and feedback to beginning teachers and create avenues for intervention with struggling veterans. • Teachers and administrators will make the appropriate referrals for intervention when they observe cases of poor performance.

26. Compared to teachers in other districts we studied, teachers in Toledo are very satisfied with the supports they receive. Toledo Districts studied in The Widget Effect *Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” **Source: Survey of 13,664 teachers in 10 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.”

27. But Toledo ranks in the middle among districts whose teachers believe the evaluation system accurately assesses them and helps them improve their instruction. Toledo Districts studied in The Widget Effect *Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” **Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.”

28. About half of Toledo teachers say there is at least one tenured teacher in their school whose instruction is poor enough to warrant dismissal. Toledo Districts studied in The Widget Effect Source: Survey of 15,176 teachers in 12 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Yes.”

29. According to data from the Toledo Federation of Teachers, the peer review program resulted in 26 departures over a four period. 6.5 tenured teachers exited annually through formal and informal means, on average. The average number of teachers in Toledo during this time was 2,084.

30. And less than half of Toledo teachers believe the district maintains high instructional standards—among the lowest percentages of the districts we studied. Toledo Districts studied in The Widget Effect *Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.” **Source: Survey of 15,176 teachers in 12 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Yes.”

31. We recommend several strategies to ensure peer review is as effective an evaluation system as possible. • Continue visits throughout a teacher’s career, whether they are referred or not. Conduct random observations to audit the quality of instruction, and refer teachers to PAR if performance is poor. • Actively canvas teachers about the quality of instruction in their team, grade level and school. Do not wait for teachers to come forward and report problems. • Apply multiple ratings rather than only satisfactory and unsatisfactory. Nuanced provides richer feedback, allows progress to be tracked, and recognizes excellence. • Use PAR for more than determining which teachers are dismissed. The most comprehensive evaluation systems drive decisions about teacher development, assignment, promotion, compensation, etc.

32. Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

33. Strong evaluation systems are based on a comprehensive set of strategies . • Evaluator Training and Support • Training and ongoing support for administrators and other evaluators to ensure consistency in evaluations. • Training on how to use evaluations to inform feedback and individualized developmental interventions. • Multiple Evaluation Criteria • Teacher contributions to student learning. • Teacher actions, as compared to clearly articulated criteria that reflect what exemplary performance looks like at a given experience level. • School and District Leader Accountability • Accountability for school and district leaders for conducting accurate, useful performance evaluations and improving teacher performance. • Processes and checks to ensure accuracy of evaluations (e.g., tenure review board). • District Policies • Shift away from centralized PD to tailored school- level development. • Requirement that teachers’ evaluations drive renewal, compensation, career path opportunities, and other rewards / consequences. • Probationary periods that last long enough to assess effectiveness comprehensively. • Reasonable documentation requirements. • Efficient processes for recognizing excellence and addressing ineffectiveness. • Evaluator Staffing • Staffing structures that allow evaluators to dedicate the time necessary to observe and evaluate teachers.

34. Teacher evaluations should assess both effectiveness and trajectory. • Effectiveness: The teacher’s positive impact on student learning during the evaluation period • Trajectory: The rate of improvement the teacher has demonstrated during the evaluation period High Reward, retain, and provide chances to lead from the classroom Renew for another year, but expect more Trajectory Recognize and appreciate, but offer targeted support Place on a rigorous improvement plan or replace Low Low High Effectiveness

35. Expectations should increase with experience, and tenure should be awarded only if a teacher meets those expectations consistently. Minimum expectations for teachers to get tenure Expectations The level of effectiveness at which the teacher’s performance could hold steady and still fully meet expectations in the future. Effectiveness Minimum expectations for first year teachers to get renewed Year 1 Year 2 Year 3 Year 4 Year 5 Duration of District’s Probationary Period (e.g., 5 years)

36. Research suggests that precise language will improve calibration of raters. “One way to increase reliability is to ensure that the evaluation instrument has clearly defined, non-subjective criteria that require minimal interpretation.” (NCCTQ, 2008)

37. Agenda About The New Teacher Project Studying Teacher Evaluation The Current Problem The Current Opportunity Preliminary Data About Peer Review Programs How to Build a Better Evaluation System Appendix

38. About The New Teacher Project The New Teacher Project (TNTP) helps school districts and states fulfill the promise of public education by ensuring that all students—especially those from high-need communities—get excellent teachers. A national nonprofit organization founded by teachers, TNTP is driven by the knowledge that although great teachers are the best solution to educational inequality, the nation’s education systems do not sufficiently prioritize the goal of effective teachers for all. In response, TNTP develops customized programs and policy interventions that enable education leaders to find, develop and keep great teachers and achieve reforms that promote effective teaching in every classroom. Since its inception in 1997, TNTP has recruited or trained approximately 33,000 teachers—mainly through its highly selective Teaching Fellows™ programs—benefiting an estimated 4.8 million students. TNTP has also released a series of acclaimed studies of the policies and practices that affect the quality of the nation’s teacher workforce, most recently including The Widget Effect: Our National Failure to Acknowledge and Act on Differences in Teacher Effectiveness (2009). Today TNTP is active in more than 25 cities, including Baltimore, Chicago, Denver, New Orleans, New York, and Oakland, among others. For more information, please visit www.tntp.org.

39. For more information: www.tntp.org