Using Rubrics to Assess Student Learning

1. Using Rubrics to Assess Student Learning Emily Langdon, PhD Coordinator for Assessment, Research, and Evaluation Division of Student Affairs Laura Martin, PhD Coordinator for Institutional Assessment Office of Institutional Assessment Anne Zanzucchi, PhD Faculty Development in Assessment Coordinator Center for Research on Teaching Excellence

2. Learning Outcomes for Today: • Participants will be… • Able to describe what a rubric is and list benefits of using rubrics • Recognize different types of rubrics • Able to describe the role rubrics play in assessment that provides information for program planning • Able to use a rubric to assess student work to gain actionable information on student learning

3. What is a Learning Outcome? • A change in attitude, aptitude or behavior that a student can describe or demonstrate after participating in a program(s) or using a service(s) (CSU Sacramento) • Ex. As a result of the health awareness workshop, students can explain how exercise affects stress. • Ex. As a result of the time management workshop, students have identified two tools to better manage their schedule.

4. What is a Learning Outcome? • Term refers both to intended and actual, observed outcomes • In doing so, it clarifies the role of LOs in program planning

5. Role of LOs in program planning Intended LO Actual LO

6. This is the “assessment cycle” Step 2 in Assessment Cycle: Conclusions Step 1 in SA Assessment Cycle: Identify LOs, POs & measures Step 2 in SA Assessment Cycle: Results

7. At what levels are LOs Developed & assessed? Institution Division of Student Affairs Academic SA Department SA Department Program Program Program Program Program Program Program Program

8. What is a rubric? How do they help us to asses student learning for use in program planning?

9. What is a rubric? • A scoring guide: • a list or chart that articulates the criteria and standards of achievement to be used to evaluate work (Suskie, 2009) • a set of criteria specifying the characteristics of an outcome and the levels of achievement for each characteristic (J. Levy, 2012)

10. What are rubrics used to score? • Assessments that require the observation of a performance or behavior(s), ex. • Presentation • Role plays • Teamwork • Performances • Assessments of written or visual artifacts, ex. • Reflection papers • Journals • Resumes • Portfolios • Art pieces

11. What is the relationship between learning outcome and a rubric? • Learning Outcome describes what students will do to demonstrate their learning, ex. • Give a presentation • Role play a…. • Work in team to… • Write a reflection that… • Rubric describes • The expected properties of that demonstration(criteria) • The possible levels of achievement/performance (standards)

12. Example learning outcome & Rubric pairing Learning Outcome: Reflecting teamwork instruction in the leadership workshop, students will employ their teamwork skills in the execution of their final group project. • Scoring Rubric: • Teamwork rubric defines four criteria and related standards for effectiveteamwork that will be examined through the interactions needed to complete the final group project: • Contributing to team meetings • Facilitating the contributions of others • Fostering constructive team climate • Responding to conflict

13. How are rubrics used in assessment as opposed to grading? • Grading : Summarizes learning demonstrated by an individual student, with feedback providing insight into and supporting his/her individual learning • Assessment:Summarizes learning demonstrated by a population of students to provide insights into how well the program is serving students as a whole • Ex. What can we learn from this? • 66% of students scored as proficient or better in basic teamwork skills • 33% scored below proficient

14. Major Types of Rubrics • Checklist • Rating Scale • Descriptive (also called analytic) • Holistic

15. General Note about Rubrics • Rubrics differ in structure - in the way criteria and performance standards are described. And thus, • In the kind of information they can provide about the quality of work • Choice of rubric depends in part upon what you want to use the information for.

16. Example of a Checklist rubric for a website (suskie, 2009)

17. Example of a Rating Scale Rubric for Evaluating Fellow Group Members (suskie, 2009) Standards

18. Example of a Descriptive (Analytic) rubric Standards Criteria

19. Example of a Holistic Rubric For Assessing Student Essay (Allen, 2004) Standards Criteria

20. Identify learning evidence to be collected and that will use a rubric to score Where do rubrics fit into cycle? Might refine rubric as part of this step Use rubric to score student work (measure learning) … Develop or identify and pilot rubric here Introduce rubric

21. Activity: Scoring student work with a rubric. Steps: • Score student work using rubric. • Review data for consistency of scoring among raters (inter-rater reliability). What if differences in scoring? • Summarize results – how best to do this? • Identify possible actions suggested by results.

22. Activity: Scoring student work with a rubric. Program Student Learning Outcome: As a result of the RA training program, RAs write concise incident reports that are professional, include critical factual details, and use language that is non-judgmental. Student Affairs Divisional Learning Outcome: Demonstrate effective written, verbal, and technological communication skills.

23. Process: Scoring student work with a rubric. • Individually: • Apply rubric to one piece of student work • Determine thescore • Provide a brief explanation for your score, referencing the rubric criteria • Provide the score to Laura

24. Review Scores: Does same piece of work receive same score for all raters? As a group: Review scores for each piece of student work. To what degree do raters agree (i.e. “inter-rater reliability”)? • Discuss reasons for scores for each students work. What reasons underpin similarities and differences? • Summarize: What if anything have we learned?

25. How deal with rater differences in real life? • Raters discuss differences and come to a shared agreement about score • Ask a third rater to score • Average rater scores if differences small (ex. < one level of performance) • Best practice: Reduce likelihood of discrepancy before scoring. “Calibrate” reviewers to apply rubric using example work so that reviewers share understanding of how to apply rubric. Revise rubric to clarify

26. How Summarize results to promote easy interpretation and, in turn, action? Most appropriate and most actionable is to summarize howfrequently we observed each level of performance • Ex. Number (or %) of student papers that were determined to be • Highly professional • Professional • Pre-professional • Unacceptable Hint: Need to have a single score for each student before start.

27. Summarize Results: Averages are problematic Averages may not capture the actual distribution of student performance. • Ex. 10 papers scored • Average score:3 = professional • Frequency of scores:

28. Interpreting results and taking action: What do results suggest to you? 1) Consider student learning results: • Are you satisfied? How do you know? If not, what might you do? To draw conclusion about success, need to have identified desired standard/level of competency/benchmark for aggregate student performance . Ex. Intend that 90% of RA incident reports will be professional or better by the end of first semester.

29. Interpreting results and taking action: What do results suggest to you? 2) Consider usefulness of rubric: • Does it work well? Could it be improved? How? 3) Consider process of data collection: • Did it work well? Could it be improved? How?

30. Resources for rubrics • Google – tons out there • Professional societies, colleagues, networks, listservs • VALUE Rubrics • Develop it yourself • Rubistar - http://rubistar.4teachers.org/

31. What makes a useful rubric? • The assignment asks students to produce work that addresses the rubric criteria (assignment, instruction, and rubric criteria are aligned). • Align rubric criteria with learning outcome and the assignment that asks students to produce the outcome. • Design rubric before giving the assignment, not after. • As possible, give rubric to students with assignment. Consider having them self-score before turning assignment in.

32. What makes a useful rubric? • Criteria and standards are sufficiently distinct and clear that raters consistently apply them in the same way (inter-rater reliability) • Use an even number of standards (ex. 4) to avoid tendency to score in middle. • The rubric is designed to yield the kind of information about student abilities needed to make planning decisions. Is it the right type of rubric? • Imagine possible results and how you might act on them. Would you have the information you need?

33. How can refine our rubric before we use it to conduct assessment? • Develop or identify rubric, shaping it to needs • Envision possible results • Pilot rubric by applying it to example work • Share rubric with students to understand how they interpret it • Revise/refine rubric to increase inter-rater reliability and usefulness to students

34. Activity: What might you use a rubric for in your own work? Reflect on your own program work in light of the Student Affairs Divisional Learning Outcomes: • Identify an opportunity to use a rubric to assess learning that would also support one of the SA Divisional Outcomes. • What would you use the rubric to evaluate? Why? • What kinds of rubric (analytic, holistic, etc.) would you choose and why?

35. What are some advantages of using rubrics to students?(suskie, 2009) • Provide to students to help them understand your expectations. Best practice – provide with the assignment • Students can apply rubrics themselves to guide learning and improvement (ex. self-score before turn in) • Improves feedback to students by identifying specific areas for improvement • Gather information that can be used to improve instruction by highlighting challenges shared by a significant proportion of students

36. What are some advantages of using rubrics for teachers and programs?(suskie, 2009) • Development of criteria and standards help to clarify vague or fuzzy goals for a program and among colleagues • Makes expectations public and shared; all colleagues can understand precisely what intend students to be able to do • Make scoring easier and faster • Increase consistency in scoring across students, among raters, and through time; allow to measure improvement • Can gather complementary indirect evidence through student self-ratings • Makes it easier to identify common strengths and weaknesses in student work, behavior, etc. to inform program planning.