Assessing using the Rubric and the Power Law

1. Assessing using the Rubric and the Power Law Anchorage - October 9, 2009

2. What is Assessment? • Anything a teacher does to gather information about a student’s level of knowledge regarding a specific topic

3. Types of assessment Three types of Assessment • Obtrusive • Unobtrusive • Student - Generated

4. Obtrusive assessment • Assessment directed/assigned by the teacher • Instruction stops to administer assessment • Assessment can take many forms; including tests, classroom activities, assignment, projects, presentations, interviews • Scores on the assessment are taken and recorded

5. Unobtrusive • Teacher observes a student performing skill or process, teacher could also use data recorded by students • Better suited to skills and processes that can be observed • Instruction does not stop in the classroom • Also occurs outside of the classroom, any place the teacher can observe a student • Scores can be recorded

6. Student Generated • Student designs assessment to demonstrate status on topic • Teacher records score and uses student demonstration as a piece of information • Student friendly version of the rubric would be an important tool

7. Formative vs. summative • Formative and summative assessments actually don’t exist! • ANY assessment can be formative or summative –DEPENDING ON HOW THE DATA IS USED! • Formative – scores taken during the course of learning to determine where a student is at that point in time • Summative – a score taken and used to describe where a student is at the end of a course of learning

8. Three Uses of Assessment • Formative Scores • Summative Scores • Instructional Feedback

9. Formative Scores • Taking and recording scores on assessments to determine where a student is at a particular point in time • Assessment can take any form • Multiple formative scores can be used to develop a learning trend and ultimately develop a summative score • Used to track student progress towards a learning goal

10. Summative Score • A score taken and recorded to measure where a student ends up on a learning goal • Typically used to determine and report a “grade” • Assessment can take any form • Should be used in conjunction with formative scores to determine summative score • Use the “Power Law” or Learning trend to determine a summative score; NOT AN AVERAGE OF SCORES!

11. Power Law Observed Score 1 1 1.5 1.5 2 1.5 3 3 2.5 Learning Trend = 2.21 2.21 2.08 2 Average Score = 1.64 1.94 1.78 1.55 1.5 Mode = 1.5 1.24 1 .71 .5 0 Pre-Test Score 2 Score 3 Score 4 Score 5 Score 6 Post-Test

12. The Power law • The power law can be applied to come up with a more accurate estimation of a student’s true score • Power law estimations are typically far closer to a student’s observed score than averaged scores • The power law is a mathematical function that takes into account the number of assessments, the score on each assessment and the time between assessments to calculate an estimated ‘true’ score • y=atb where y is a score on a particular assessment, t is the time at which the assessment was administered and a and b are constants

13. How is the power law used? • Electronic grade books can be designed to automatically calculate it • Teachers can use their professional judgment by collecting data and looking for the pattern in the data. The data points will probably fit a power law curve over time. • For example, a student (using the rubric), gets the following scores • 0.0, 1.0, 2.0, 2.5, 2.5, 2.5, 3.0, 3.0, 3.5, 3.0 • What would the student’s estimated true score be? • Averaging results in a score of 2.3 – the student is penalized for not knowing the information at the beginning ”

14. Using Teacher Judgment to Determine a student’s grade • Imagine that a student has received the following scores on a measurement topic: • 1.0, 1.0, 1.5, 1.5, 2.0, 1.5, 3.0 • What grade would they receive? • In your professional judgment – • Do they deserve a 3.0? Why or why not? • Do they deserve a 2.0? Why or why not? • Considerations: Look at the trend in the data – is it going up? Have they demonstrated consistent success at any level? Do you believe they can accomplish a specific level? Do you need more data?

15. How many data points do I need? • The rubrics are designed so that a teacher can use fewer data points based on a set criteria to estimate the “true score” • The short answer is that you need as many as it takes to get a good picture; using professional judgment, assessment data, and your knowledge of the student; of what a student knows at any given period of time • 4- 5 are ideal. The less certain you are about a student’s “true score”, the more data you need

16. Implications • Zeros given on an assignment or assessment because the student did not do it skew the calculation of a true score • If you are trying to measure what a student knows and is able to do, use other means to measure and report work completion, behavior etc. • A separate set of rubrics, or a separate grade can be used

17. Instructional Feedback • Data from assessments used to determine and adjust course of instruction • Represents a student’s or class’s understanding or progress towards a learning goal at any point in time • Typically not recorded or scored

18. Designing Assessments - Obtrusive • Assessments can be designed to assess all levels of the rubrics at the same time. • Advantage – Students know what the ultimate goal is • Disadvantage – Students begin a course of learning with very low scores, but improve from there • Assessments could be designed to assess level 2.0 material first, and once mastered, move on the level 3.0 • Advantage – Opportunity for individualized instruction, foundational material mastered before moving onto learning goal • Disadvantage – If used for a whole class, may hold students back that are ready to move on. Students may not have a clear picture of the ultimate goal

19. Designing assessments - unobtrusive • Create situations that allow the teacher to observe student performing skills or processes • “Catch” students performing the skill during the course of instruction • Record data just like any other assignment

20. Designing assessments – Student generated • Allow and encourage students to come up with their own way to demonstrate progress • Record data and use to help determine progress • Write a student friendly version of the rubric • Place responsibility on students to provide data or “prove” their progress to teacher