Project on Educator Effectiveness & Quality

1. Project on Educator Effectiveness & Quality Chancellor Summit September 27, 2011 Cynthia Osborne, Ph.D.

2. Accountability System for Educator Preparation Programs (SB 174) • Standard 1: Certification Exam Passing Rates • Standard 2: Principal Survey of 1st Year Teacher; Graduates’ Survey of Preparedness • Standard 3: Teachers’ Influence on Growth in Student Achievement in First Three Years Post Certification • Standard 4: Quality of the Field Supervision

3. Purpose of PEEQ • Tasked by TEA to design a metric to determine EPP’s graduates’ influences on student achievement • Metric is required due to: • SB 174 student achievement standard (#3) • State longitudinal data system grant (SLDS II) • School improvement grant (SIG) • State fiscal sustainability fund (SFSF) • Will focus only on 1st through 3rd year teachers • Objective is to provide feedback to EPPs to improve teaching and student performance – not just for accountability

4. SB 174 and PEEQ SB 174 Standard III Please note that the weights of SB174 and the weights of Standard III have not been determined.

5. What Predicts Effective Teaching? • Years of experience: teachers show 2/3 of all improvement in first 5 years • General intelligence: selectivity of the college, college entrance exams, teacher licensing exams, and IQ tests • Certification in subject area (especially math): masters degree is not predictive • Rigorous observations of teachers in classroom • Most variation in teacher effectiveness is not explained by “teacher inputs” – focus has shifted to “student outcomes”

6. Measuring Teacher Effectiveness in Texas • Our current system, PDAS, does not provide information to the state or EPPs on effective teachers • Over 98% of teachers are rated proficient, thus EPPs have no indicators to improve programming • NCLB standards (highly effective) are inconsistent with PDAS findings; students are not all proficient • Goal is to improve the quality of the pipeline of new teachers in the state by providing EPPs with information on how their graduates are influencing student achievement

7. Measuring Teacher Effectiveness Across the Nation • States and districts are increasingly using student performance to measure teacher effectiveness • Have moved from a system of measuring “teacher inputs” to measuring “student outcomes” • 31 states have initiated plans • States are moving quickly from plan to implementation; LEAs move more slowly • Wide variety exists in components measured and instruments used • Few states use teacher effectiveness for EPP accreditation – although momentum is gaining

8. An Effective Metric Should Be: • Comprehensive: Includes multiple sources; not just test scores • Useful: Leads to improvement in EPPs and teaching • Valid & Reliable:Reflects student achievement and considers variations in populations of students; Problems with measurement error, test scaling, and data quality must be minimized • Integrated: EPPs held accountable for training teachers to do what teachers will be held accountable for doing • Transparent: Clear how components are measured and weighted

9. Decisions PEEQ Must Make • What components should be included in a comprehensive metric to determine a teacher’s effect on student achievement? How should we weight the components? • What is the most valid and reliable method to measure student performance using test scores?

10. A Comprehensive Metric Should Include: • Growth in Student Performance on TAKS (VAM) • Observations of Teachers in Classroom • Classroom Environment • Curriculum and Instruction • Principals’ Assessment of Influence on Achievement • Teacher’s Impact on Student Achievement • School or Grade Level Growth in Performance

11. Weighting of Components Will Be Determined Empirically

12. PEEQ’s Metric

13. Measuring Student Performance of EPPs’ Graduates • Common approaches • Percent of students passing standardized test • Percent increase in students passing test • Strengths • Straight forward and transparent • Common standard and expectation of knowledge • Limitations • Does not account for a student’s prior knowledge or growth • May lead to focus on teaching to test

14. Measuring Student Performance of EPPs’ Graduates • Value Added Models • Goal is to predict what a student should score and compare to actual performance – “teacher effectiveness” is the difference • Various methods used to calculate • Require a large enough sample to create comparisons • Need sufficient data to accurately predict student performance • Attribution of students to teachers is crucial • Tests must be vertically aligned to show growth • PEEQ will use a VAM, but not at individual teacher level, yet

15. Measuring Student Performance of EPPs’ Graduates • Value Added Models (Continued) • Strengths • Account for past performance of students and student - and school-level characteristics that may affect teaching and student performance • Emphasis is on growth in student performance • Teachers’ past effectiveness is highly predictive of future • Limitations • Difficult to understand calculations (less transparent) • Measurement error may bias estimates

16. Observations of Teachers • Must be mindful of the scale: • 75,000 beginning teachers (in first 3 years) • 1,400 school districts • Principals are required to observe beginning teachers and fill out a survey on each teacher (ASEP) • This survey provides the platform to gather information on teachers’ classroom environment and instruction (elements linked to student achievement)

17. Classroom EnvironmentTo what extent is this beginning teacher able to . . . (4 pt scale) • Effectively implement discipline-management procedures? • Communicate clear expectations for achievement and behavior that promote and encourage self-discipline and self-directed learning? • Provide support to achieve a positive, equitable, and engaging learning environment? • Build and maintain positive rapport with students? • Build and maintain positive rapport and two-way communication with students’ families?

18. InstructionTo what extent is this beginning teacher able to . . . (4 pt scale) • implement varied instruction that integrates critical thinking, inquiry, and problem solving? • respond to the needs of students by being flexible in instructional approach and differentiating instruction? • use the results of formative assessment data to guide instruction? • engage and motivate students through learner-centered instruction? • integrate effective modeling, questioning, and self-reflection (self-assessment) strategies into instruction? • assume various roles in the instructional process (e.g. instructor, facilitator, audience)? • set clear learning goals and align instruction with standards-based content? • provide quality and timely feedback to students?

19. Principal Assessment • “How would you rate this teacher’s influence on student achievement?” • 10 point scale: unacceptable to exceptional

20. Information to EPPs for Standard #3 • EPPs will get a total score, and disaggregated information on graduates • Information to EPPs will include: • Overall effectiveness • Aggregate scores by each metric component • Scores by: • Certification program • Type/location of school • Grade level/subject matter

21. Feedback from Districts/Teachers • Want better indicators of effectiveness; have concerns about the uses of the metric • Claim that quality of field supervision distinguishes EPPs • Encourage a stronger dialog and more interaction between districts and EPPs to align teaching and practice

22. Feedback from EPPs • Deans want better indicators of effectiveness of graduates • Concerned about variation in support teachers receive across districts • Concerned that information has not previously been available; would like information for two years prior to accountability • No consensus on what new teachers should know relative to experienced teachers

23. We want your input! • We have a statistical advisory group and invite each EPP to designate a member. • We conduct quarterly stakeholder meetings and invite all stakeholders to participate. • We update our website regularly with new research briefs: www.utexas.edu/lbj/peeq

24. Expected Timeline • Principal Survey (Standard #2) administered in summer 2011 • Data linking teachers and students available fall 2011 • Pilot Metric available spring 2012 • Pilot Metric presented to SBEC June 2012