1 / 43

EXPLORING INTER-RATER RELIABILITY

EXPLORING INTER-RATER RELIABILITY. Professional Learning Team Conference June 2014. Desired Outcomes:. Each participant will leave the session knowledgeable about inter-rater reliability What it is Why it’s important, and

joanna
Download Presentation

EXPLORING INTER-RATER RELIABILITY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. EXPLORING INTER-RATER RELIABILITY Professional Learning Team Conference June 2014

  2. Desired Outcomes: Each participant will leave the session knowledgeable about inter-rater reliability What it is Why it’s important, and How to use evidence to consistently and accurately rate teachers’ performance

  3. Goals for Inter-Rater Reliability 2014-2015 • Demonstrate familiarity with district rubric • Describe proficient practices based on district rubric descriptors • Provide feedback to educators based on evidence collection • Calibration of evaluators

  4. GOALS: PARTICIPANTS WILL: • Establish a common vocabulary around inter-rater reliability • Be able to explain levels of performance with look-fors • Gain a deeper understanding of proficient level of performance aligned to standards

  5. CHINESE PROVERB: • What I hear, I forget • 5% retention after 24 hours • What I see, I remember • 30% retention rate if use of audio-visual and demonstration • What I do, I understand • 80% retention rate with discussion and • practice by engagement

  6. NORMS Share experiences to enrich others Ask questions Learn by doing Set aside any preconceived notions about evaluating educators Apply your own work

  7. LEARNING TARGET: Participants will become familiar with look fors of effective practice and be able to use that knowledge to assess performance in three focused areas: • Student Engagement • Questions, Prompts & Discussion • Assessment

  8. EVALUATION: NATIONAL FOCUS ON TEACHER EFFECTIVENESS AND EVALUATION However, most schools fail to evaluate teachers in a meaningful way, with 94% of teachers receiving the highest ratings. Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf. EFFECTIVENESS: Teacher effectiveness is the single most important school-related factor affecting student learning. Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf

  9. INTER-RATER RELIABILITY: a systemic approach to calibrate observations for consistent and fair professional practice ratings

  10. Learning Experience: At your table define and describe Student Engagement Capture your collective thoughts on chart paper

  11. Educational author and former teacher, Michael Schmoker shares in his book, Results Now, a study that found of 1,500 classrooms visited, 85 percent of them had engaged less than 50 percent of the students In other words, only 15 percent of the classrooms had more than half of the class at least paying attention to the lesson

  12. WHAT DO "ENGAGED" STUDENTS LOOK LIKE? You will see students... • Paying attention (alert, tracking with their eyes) • Taking notes (particularly Cornell) • Listening (as opposed to chatting, or sleeping) • Asking questions (content related, or in a game) • Responding to questions (whole group, small group, four corners, Socratic Seminar) • Following requests (participating, Total Physical Reponses (TPR), storytelling, Simon Says) • Reacting (laughing, crying, shouting, etc.) • Reading critically (with pen in hand • Interacting with other students Michael Schmoker

  13. ENGAGING STUDENTS IN LEARNING • Activities aligned with the goals of the lesson • Student enthusiasm, interest, thinking, problem-solving… • Learning tasks that require high-level student thinking and are aligned with lesson objectives • Students are highly motivated to work on all tasks and are persistent even when the tasks are challenging • Students actively "working," rather than watching while the teacher "works“ • Suitable pacing of the lesson: neither dragging nor rushed, with time for closure and student reflection

  14. STUDENT ENGAGEMENT When determining the level of performance for Engaging Students in Learning, consider the quality of activities, assignments, and resources the instructor uses to pull students into the lesson It is also important to understand how the lesson uses student grouping and pacingwhen determining the level of performance for this component

  15. Learning Experience: At your table define and describe Assessment Capture your collective thoughts on chart paper

  16. THE PURPOSE OF  CLASSROOM ASSESSMENT is to support student learning and to communicate that learning to others

  17. ASSESSMENT: In order to support student learning classroom assessment needs to: • involve students deeply in the assessment process • provide specific, descriptive feedback during the learning, and • include evaluative feedback as required to communicate and report progress over time

  18. ASSESSMENT: To ensure success of all: • students need to know what they already know • know what needs to be learned and • know what success looks like Students also need to learn how to guide their own learningthrough being involved in setting and using criteria, giving themselves feedback for learning (self-assessment), setting goals, collecting evidence and communicating that evidence of learning to others

  19. INDICATORS THAT MAY BE USED AS A GUIDE FOR LOOKING FOR EVIDENCE OF ASSESSMENT • The teacher pays close attention to evidence of student understanding • The teacher poses questions specifically created to elicit evidence of student understanding • The teacher circulates to monitor student learning and to offer feedback • Students assesses their own work against established criteria • The teacher adjusts instruction in response to evidence of student understanding (or lack of it)

  20. BREAK

  21. Learning Experience: Work with your District partners to identify the elements in your rubric that focus on • Student Engagement • Question, Prompts & Discussion • Assessment

  22. Learning Experience: While observing the video, collect evidence that you will later use to rate the teacher’s performance using your district rubric

  23. Remember… You will use your district’s rubric to rate the teacher’s performance in the following areas: Student Engagement Question, Prompts & Discussion Assessment

  24. Math Lesson: CC Standard 7.G.6 https://www.teachingchannel.org/videos/surface-area-lesson# Video

  25. LUNCH

  26. In just a moment you will be asked to review the data you’ve collected, cluster and code it for easy interpretation, and rate the teacher’s performance using your district’s rubric

  27. DATA Evidence DATA Interpretation DATA Judgment

  28. WHAT IT LOOKS LIKE TOCLUSTER AND CODE

  29. ANOTHER METHOD OF EXAMINING YOUR EVIDENCE

  30. ANOTHER METHOD OF EXAMINING YOUR EVIDENCE

  31. With an elbow partner, use the verbatim transcript of the teacher’s observation to identify and organize evidence to accurately and fairly rate the teacher’s performance in the areas of • Student engagement • Questions, Prompts and Discussion • Assessment

  32. Look at your evidence for each component.  Where does the preponderance of evidence align with levels of performance? • Match your data with the rubric performance level description • Assign a performance level

  33. Use your clicker to rate this teacher’s performance using the numbers Proficient Unsatisfactory • Basic Distinguish

  34. TESTIDENTIFY YOUR FAVORITE CANDY BAR • Snickers • Milky Way • Kit Kat • Trix :01

  35. STUDENT ENGAGEMENT • Unsatisfactory • Basic • Proficient • Distinguish :01

  36. DISCUSS

  37. QUESTIONS, PROMPTS & DISCUSSION • Unsatisfactory • Basic • Proficient • Distinguish :01

  38. DISCUSS

  39. ASSESSMENT • Unsatisfactory • Basic • Proficient • Distinguish :01

  40. DISCUSS

  41. ONE – TWO PUNCH • One thing you heard today that you want to remember • Two things you heard that you want to act on before the next PLT Conference John Doe 1234 Paradise Lane Anywhere, OR 97890

  42. QUESTIONS

  43. Thank you for your active participation and cooperation 

More Related