Increasing effective and highly effective teachers State Considerations

1. Lynn Holdheide, Vanderbilt University Heather Buzick, Ph. D., Educational Testing Service Samantha Warburton, Massachusetts Dept of Education 1 • Increasing effective and highly effective teachers State Considerations in Designing and Implementing Evaluation Systems that Include Teachers of Students with Disabilities

2. Multiple sources of evidence inform the summative performance rating 2 Massachusetts Department of Elementary and Secondary Education

3. State Considerations in Designing and Implementing Evaluation Systems that include Teachers of Students with Disabilities Lynn Holdheide Vanderbilt University, National Comprehensive Center for Teacher Quality Office of Special Education Programs Project Director’s Conference Tuesday, July 24, 2012 10:30 a.m. – 12:00 p.m.

4. Learning Targets • Participants will have an increased awareness of: • considerations states should contemplate when designing teacher evaluation systems; • potential solutions to mitigate perceived challenges; • practical state approaches in teacher evaluation design; and, • available research to guide decisions and needed research to validate state and district efforts.

5. Federal and State Theory of Action 5

6. The Goal of Teacher Evaluation

7. Policy Requirements 7 • Increasing effective and highly effective teachers • number and/or percentage • retention and equitable distribution • Method for determining and identifying effective and highly effective teachers • must include multiple measures • Effectiveness evaluated, in significant part, on the basis of student growth • supplemental measures may include, e.g. multiple observation based instruments

8. Considerations for States when Evaluating Teachers of Students with Disabilities • Did not state special education teachers Rationale: More than special education teachers are responsible for the academic progress of students with disabilities. • Few new evaluation models differentiate the process Rationale: Perceived fairness, improved implementation fidelity, decreased costs. • Intent to help teachers improve Rationale: evaluation process accurately measures growth and reinforces the use of high-leverage instructional principles

9. 4 State Considerations when Evaluating Teachers of Students with Disabilities • Ensuring needs of students with disabilities and their teachers are considered at the beginning of the design process. • Central to ensuring that the evaluation process leads to quality feedback regarding teacher performance. • Design is universal and not retro-fitted after the fact.

10. 4 State Considerations when Evaluating Teachers of Students with Disabilities 2) Measuring growth of students with disabilities • Electing to exclude the scores of students with disabilities within value-added modeling or other measures of student learning could greatly limit teacher accountability. • When growth is not accurately measured for students with disabilities or performance expectations are not aligned with possible learning outcomes, teachers may be less likely to accept students with disabilities in their classrooms.

11. A Forum of State Special Education and Teacher Effectiveness Experts and Researchers • To identify the challenges in using the growth of students with disabilities to evaluate educators • To develop considerations for states when designing systems that include the academic growth of students with disabilities • To identify needed areas of research http://www.tqsource.org/pdfs/TQ_Forum_SummaryUsing_Student_Growth.pdf

12. Measuring Student Growth • Teachers want to be confident that the measures used are a fair and accurate representation of both student growth and their contribution to that growth.

13. Tested Subjects 13 • Challenges in Using Growth Models for Special Educators & SWD • A research-derived value-added model for special educators does not exist • Student learning trajectory • Students assessed with accommodations • Small student samples commonly associated with special education caseloads • Student mobility • Test Scaling

14. Considerations for States • Use multiple measures and consider weights to reflect the amount of evidence in support of validity and accuracy for value-added scores • Based on transparent judgment initially; then empirically based • Support accessible assessments that offer precise measurement along the entire score scale (e.g., multistage adaptive assessment, universal design) • Create a standardized system to accurately assign, monitor, and record the use of testing accommodations • Adopt a roster validation system; use full roster method to give 100 percent credit to all teachers in a coteaching situation

15. Measuring Teachers’ Contributions to Student Learning Growth: A Summary of Current Models

16. Potential Challenges for Students With Disabilities in the SLO Process • Students with disabilities could be overlooked in the SLO process. Therefore, the growth (or lack thereof) could go unnoticed. • Increased need for teacher capacity to collect, interpret, and monitor student performance data against standards-aligned, rigorous goals. • The comparability of measurement and student growth is compromised  because the process may not be standardized or objective.

17. Sampling of State Considerations • Ensure that growth for all students, including students with disabilities, is accounted for in the SLO process. • Encourage collaboration between general and special education teachers to construct SLOs to ensure alignment with the established standards AND to accommodate the specific learning needs. • Encourage that SLOs can be tiered so that student targets can be differentiated according to the present levels of student performance.

18. Rhode Island Student Learning Objectives • There is a requirement that all students are covered under an SLO: • General education teachers are responsible for the progress and mastery of all students on their rosters, including students with disabilities. • Teachers are encouraged to set tiered goals so that targets are differentiated. • General education and special education teachers are encouraged to work collaboratively

19. Rhode Island Student Learning Objectives • Partnered with special education teachers in early adopter districts and local institutions of higher education to draft sample SLOs. • Example SLOs for students with disabilities are located at http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/SLO.aspx. • Guidance document that provides recommendations on SLO development across context http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/Docs/SPED_FAQ_revised.pd

20. Groupwide Value-Added Challenges • Teachers may be held accountable for students they have never taught or had the opportunity to influence. • Accountability for the growth (or lack thereof) of students with disabilities may not be captured or monitored if students with disabilities are not included in the value-added scores. • Teachers may not be as motivated to improve student mastery of state standards if there is no direct accountability for their content areas.

21. 4 State Considerations when Evaluating Teachers of Students with Disabilities • The appropriateness of existing measures of instructional practice • Evidence-based instructional practices for students with disabilities • direct/explicit instruction, scientifically based reading instruction, learning strategy instruction • Specific roles and responsibilities of special educators • Consultant, expert, and/or teacher • IEPfacilitation, collaboration, secondary transition, social and behavioral interventions, compliance with legal mandates • Specific curricular needs • Expanded Core Curriculum (Post School Outcomes) • Access

22. 4 State Considerations when Evaluating Teachers of Students with Disabilities • The distinct considerations for teachers (both general and special education) serving in a coteaching capacity. • Should effective co-teaching practices be a factor in teacher evaluation? • Would using the general observation rubric suffice? • How should student growth be attributed in an co-teaching classroom?

23. Lynn Holdheide • Senior TA Consultant • lynn.holdheide@vanderbilt.edu • 615-477-1880 • 1000 Thomas Jefferson Street NW • Washington, DC 20007-3835 • Phone: 877-322-8700 or 202-223-6690 • Website: www.tqsource.org

24. State Considerations in Designing and Implementing Evaluation Systems that include Teachers of Students with Disabilities Heather Buzick Educational Testing Service OSEP Project Directors' Conference July, 2012

25. Overview • Measurement challenges, current research, suggestions for practice • Student academic growth as an indicator of teacher effectiveness (value-added) • Teacher observation protocols • Multiple teachers • Research ideas

26. Concerns • Teachers are not held accountable for the education of all students if indicators from students with disabilities are not included in teacher evaluation systems • Current evaluation systems may not capture differentiated instruction • Challenges can create disincentives to accept particular students into the classroom • Perceived unfairness to teachers with high numbers of students with disabilities

27. Two broad areas that can impact the meaning of value-added scores • Measurement challenges • Various threats to the validity of inferences about student academic growth that is attributed to teachers • Complex instructional context • Can impact evaluation of both general education teachers and special education teachers

28. Measurement challenges (I) • Testing accommodation use • Inconsistent use across years • Particularly for those associated with a score boost (e.g., read aloud, extended time) • Extreme low performance on linear state assessments • Difficult to get a good measure of growth • Systematic and predictable

29. Measurement challenges (II) • Small samples or missing data • Some special education classrooms • Lower match rates for students with disabilities due to mobility, absence on test dates, etc.

30. Instructional context • Shared responsibility between general education and special education teachers • Time that students spend in the regular classroom learning content • The performance of all students in a classroom may be impacted – positively or negatively – by the presence of a co-teacher, extra funding support for special services, peer behaviors, or other factors not directly related to an individual teacher

31. Evidence and research Analyses with state general assessment data to: • Identify systematic characteristics of student data that may impact meaning and validity of inferences about teachers • Research robustness of value-added models to systematic characteristics • Document factors that do not threaten validity • Find solutions for those that do

32. Classroom context Example from one state database, general assessment reading and math scores and associated teacher: • 59% of teachers in grades 3-8 in the sample had at least one student with a disability in the classroom • General education teachers had 3 to 4 students with disabilities on average or 16% of students in classroom • 5 students on average for special education teachers (13% of teachers)

34. Accommodations • 56% of teachers in the sample had at least one student who used an accommodation on the general assessment • Half of those teachers has at least one student with inconsistent accommodation use across years • An average of 3 students per teacher used accommodations inconsistently • Across three states, up to ¼ of students who received accommodations on the general assessment did so inconsistently across years • Related to grade

36. Considerations for states • Identify any systematic and predictable factors that may impact teacher ratings based on student outcomes (annually, across state) • Consider including factors in value-added model • Document and communicate factors that do not threaten the validity of inferences about teacher effectiveness • Identify special cases where validity is in question • For special cases • Work with teachers to understand the quality of their individual value-added score given their particular classroom context • Adjust weights on value-added scores and other measures

37. Observation Protocols • Single general protocol (e.g., Danielson’s Framework for Teaching) • May not sufficiently outline expectations for instruction provided to students with disabilities • May not create incentives for teachers to adopt effective practices for teaching students with disabilities • Definition of effective teaching does not always map on to definitions of effective teaching put forward by special education community • Some teachers may never able to be rated in top category

38. Reliability of observers’ scores • Can observers reliably differentiate between teachers who do and do not make use of effective instructional practices for students with disabilities? • Rater background and familiarity with educating students with disabilities can contribute variability to observers’ scores

39. Suggestions - Observations • One option for districts would be to adopt observation protocols designed specifically for use with students with disabilities • Classroom Climate Scale, as developed by McIntosh, Vaughn, Schumm, Haager, & Lee (2002) • Individualizing Student Instruction protocol (Connor et al., 2009), which examines how teachers’ tailor their instruction to students’ individual needs • This option is costly and unlikely to be adopted by states or districts Connor, C. M., Morrison, F. J., Fishman, B. J., Ponitz, C. C., Glasney, S., Underwood, P. S., et al. (2009). The ISI Classroom Observation System: Examining the literacy instruction provided to individual students. Educational Researcher, 38, 85–99. McIntosh, R., Vaughn, S., Schumm, J. S., Haager, D., & Lee, O. (1993). Observations of students with learning disabilities in general education classrooms. Exceptional Children, 60, 249-261.

40. Suggestions - Observations • A more feasible option is to supplement an existing observation protocol with a subset of items specific to teaching students with disabilities • Consider incorporating domains from one of the existing protocols specific to students with disabilities • Alternatively, existing response categories on observation protocols could be adapted to more appropriately reflect teachers’ interactions with students with disabilities • One viable short-term solution would be to develop “scoring support documents” to assist observers in the scoring process, with an emphasis on the kinds of evidence-based practices that have proven to be effective for teaching students with disabilities

41. Suggestions – Observer ratings • To improve observer familiarity with instruction for students with disabilities, districts could ensure that observers have some training or background specific to special student populations • During rater training – include one video of special education instruction

42. Considerations formultiple teachers • A roster validation system can increase both the face validity of value-added scores as well as the accuracy of estimates (Hock & Isenberg, 2011) • Example: The Houston Independent School District uses a system where teachers can regularly log in and verify the accuracy of their rosters

43. Considerations formultiple teachers • Full roster method • Both the general education and special education teachers receives 100 percent responsibility of their shared students • Helps to ensure that students with disabilities are not viewed as the sole responsibility of the special education teacher (Hock & Isenberg, 2011) Hock, H., & Isenberg, E. (2011). Methods for accounting for co-teaching in value-added models. Princeton, NJ: Mathematica Policy Research. Retrieved from http://www.aefpweb.org/sites/default/files/webform/Hock-Isenberg%20Co-Teaching%20in%20VAMs.pdf

44. Research ideas • Research on value-added models: • Routine validation research that includes scores from students with disabilities • Sensitivity studies with specific variables relevant to students with disabilities (i.e., accommodation use, entry/exit from special education) • Explore the correspondence between value-added scores and other indicators of teacher effectiveness by subgroup

45. Research ideas • Research on observation protocols: • Provide guidance on how to modify rubrics to include items or response categories specific to students with disabilities • Conduct research on validity and reliability for modified rubrics or specific observation protocols for students with disabilities • Evaluate observer performance specific to those who educate students with disabilities

46. State Considerations in Designing and Implementing Evaluation Systems that Include Teachers of Students with Disabilities OSEP Program Directors’ Conference July 24, 2012 Samantha Warburton Educator Evaluation Project Lead

47. One State’s Perspective… When policy and practice must move faster than research and development, where do you begin? Massachusetts philosophy: • Don’t let perfection become the enemy of good: the work is too important to delay • Understand this is just the beginning: we will be able to do this work with increasing sophistication each year • Phase-in implementation: take advantage of emerging research, resources, and feedback from the field 47 Massachusetts Department of Elementary and Secondary Education

48. MA Educator Evaluation Framework: Status and Timeline 48 • June 2011 – MA Board of Education passed new regulations • September 2011 – Implementation began in 34 “Level 4” schools, 11 “Early Adopter” districts, and 4 Special Education Collaboratives • January 2012 – MA Dept of Elementary and Secondary Education (ESE) published the Massachusetts Model System for Educator Evaluation • September 2012 – Implementation begins in all RTTT districts • September 2013 – All districts implement educator evaluation • September 2013 – Districts begin phase-in of Rating of Impact on Student Learning Massachusetts Department of Elementary and Secondary Education

49. Educators earn two separate ratings * Educator Plans above apply only to experienced educators; all new educators are placed on a “Developing Educator Plan.” Massachusetts Department of Elementary and Secondary Education

50. 5 Step Evaluation Cycle • Every educator is an active participant in an evaluation • Process promotes collaboration and continuous learning • Process applies to all educators Massachusetts Department of Elementary and Secondary Education