June 20, 2011 Measures of Effective Teaching (MET)project
z What does it take to build better feedback & evaluation systems? (and why start there, anyway?)
teaching effectiveness close the gap
“we can’t fire our way to Finland” - Linda darling hammond
Trustworthiness Tests • 1. Face Validity • Do teachers recognize the observation instrument and other measures as reflecting qualities of practice they value? • 2. Coherence • Do the measures match your district’s theory of instruction? • 3. Predictive Validity • Do scores on your measures correlate with outcomes that you value, such as gains in student learning? • 4. Scoring Reliability • If a different rater had been assigned to the observation or the assessment, would the score be the same?
The Measures of Effective Teaching Project Participating Teachers • Two school years: 2009–10 and 2010–11 • >100,000 students • Grades 4–8: ELA and Math • High School: ELA I, Algebra I and Biology
Research Partners • Our primary collaborator include: • Mark Atkinson, Teachscape • Nancy Caldwell, Westat • Ron Ferguson, Harvard University • Drew Gitomer, Educational Testing Service • Eric Hirsch, New Teacher Center • Dan McCaffrey, RAND • Roy Pea, Stanford University • Geoffrey Phelps, Educational Testing Service • Rob Ramsdell, Cambridge Education • Doug Staiger, Dartmouth College • Other key contributors include: • Joan Auchter, National Board for Professional Teaching Standards • Charlotte Danielson, The Danielson Group • Pam Grossman, Stanford University • Bridget Hamre, University of Virginia • Heather Hill, Harvard University • Sabrina Laine, American Institutes for Research • Catherine McClellan, Educational Testing Service • Denis Newman, Empirical Education • Raymond Pecheone, Stanford University • Robert Pianta, University of Virginia • Morgan Polikoff, University of Southern California • Steve Raudenbush, University of Chicago • John Winn, National Math and Science Initiative The MET ProjectThe Bill & Melinda Gates Foundation launched the Measures of Effective Teaching (MET) project in fall 2009 to test new approaches to measuring effective teaching. The project’s goal is to help build fair and reliable systems for teacher observation and feedback to help teachers improve and administrators make better personnel decisions. With funding from the foundation, the data collection and analysis are being led by researchers from academic institutions, nonprofit organizations, and several private firms and are being carried out in seven urban school districts.
The MET Project research How well can … • Student Perception Surveys • Structured Classroom Observations + • Pedagogical Content Knowledge Test + • Student Outcomes + … produce valid & reliable measures of teaching?
Classroom Observation Using Digital Video
Value-added on each test • Value-added on state assessment • Value-added on supplemental assessments
Validation Engine • Software provides analysis of: • Rater consistency • Rubric’s relation to student learning Raters score MET videos of instruction System picks observation rubric & trains raters
Ensuring Reliable Observations Rater Training & Consistency: building trust
Facilitating Feedback Direct Feedback Real time (ear-bud) coaching Remote Collaboration
Preliminary Finding #1 the kids know
Student Perceptions Captivate Consolidate Challenge Confer Control Clarify Care TestPrep Care • My teacher makes me feel that s/he really cares about me • My teacher seems to know if something is bothering me • My teacher really tries to understand how students feel about things • My teacher takes the time to summarize what we learn each day • The comments that I get on my work in this class help me understand how to improve • If you don’t understand something , my teacher explains it a different way. • My teacher knows when the class understands, and when we do not. • My teacher has several good ways to explain each topic that we cover in the class. • My teacher asks students to explain more about the answers they give. • My teacher doesn’t let people give up when the work gets hard. • In this class, we learn to correct our mistakes. • Students in this class treat the teacher with respect • My classmates behave the way the teacher wants them to • Our class stays busy and doesn’t waste time • My teacher makes learning enjoyable • My teacher makes learning interesting • I like the way we learn in this class • My teacher wants us to share our thoughts • Students get to decide how activities are done in this class • I have learned a lot this year about [the state test] • Getting ready for [the state ] test takes a lot of time in our class Control Clarify Challenge Captivate Confer Consolidate TestPrep
Students Distinguish Between TeachersPercentage of Students by Classroom Agreeing
Student Perceptions • Top 5 Correlations Category Rank Survey Statement • Students in this class treat the teacher with respect Control 1 • My classmates behave the way my teacher wants them to Control 2 3 • Our class stays busy and doesn’t waste time Control 4 • In this class, we learn a lot every day Challenge 5 • In this class, we learn to correct our mistakes Challenge • I have learned a lot this year about [the state test] Test Prep 33 Test Prep 34 • Getting ready for [the state test] takes a lot of time in our class
Preliminary Finding #2 the consequences are great
Video validation Developer Instrument University of Virginia • Classroom Assessment Scoring System, CLASS • Mathematical Quality of Instruction (MQI) University of Michigan • Framework for Teaching Charlotte Danielson • Quality Science Teaching (QST) Stanford University • Protocol for Language Arts Teaching Observation (PLAT0) Pam Grossman • National Board for Professional Teaching Standards NBPTS Natl Math & Sci Initiative • UTeach Observation Protocol (UTOP)
MET Logical Sequence Measures reliable Measures predict Research Use Measures stable under pressure Measures combine ? Effective Teaching Index Measures fairly reflect teacher Teaching Effectiveness Dashboard Measures improve effectiveness