Essential Practices for Effective Rubric Implementation:

1. Essential Practices for Effective Rubric Implementation: Tips on Training and Using Data for Improvement Linda Siefert, Ed.D Director of Assessment University of North Carolina-Wilmington Ashley Finley, Ph.D Senior Dir. of Assessment & Research, AAC&U National Evaluator, Bringing Theory to Practice General Education & Assessment Conference Portland, OR February 27, 2014

2. Capturing What Matters: VALUE Rubrics Initiative • Rubric Development • 16 rubrics • Created primarily by teams of faculty • Inter-disciplinary, inter-institutional • Three rounds of testing and revision on campuses with samples of student work • Intended to be modified at campus-level • Utility • Assessment of students’ demonstrated performance and capacity for improvement • Faculty-owned and institutionally shared • Used for students’ self-assessment of learning • Increase transparency of what matters to institutions for student learning

3. VALUE Rubrics (www.aacu.org/value) • Knowledge of Human Cultures & the Physical & Natural Worlds • Content Areas No Rubrics • Intellectual and Practical Skills • Inquiry & Analysis • Critical Thinking • Creative Thinking • Written Communication • Oral Communication • Reading • Quantitative Literacy • Information Literacy • Teamwork • Problem-solving • Personal & Social Responsibility • Civic Knowledge & Engagement • Intercultural Knowledge & Competence • Ethical Reasoning • Foundations & Skills for Lifelong Learning • Global Learning • Integrative & Applied Learning • Integrative & Applied Learning

5. The Anatomy of a VALUE Rubric Criteria Levels Performance Descriptors

6. The Role of Calibration • Essential training through collaboration and discussion • Transparency of standards (without standardization) • Building shared, inter-disciplinary knowledge around skills assessment and application of the rubric • Shared stake in data collection and discussion for improvement • Building intentionality of assignment development

7. The Calibration Training Process • Scoring Steps: • Review rubric to familiarize yourself with structure, language, performance levels • Ask questions about the rubric for clarification or to get input from others regarding interpretation • Read student work sample • Connect specific points of evidence in work sample with each criterion at the appropriate performance level (if applicable) • Calibration Steps: • Review scores • Discuss scoring rationale • Opportunity to change scores

8. The Ground Rules • We are not changing the rubric (today). • This is not grading. • Think globally about student work and about the learning skill. Think beyond specific disciplinary lenses or content. • Start with 4 and work backwards. • Connect evidence in work sample with language in performance cell. • Pick one performance benchmark per criterion. Do not use “.5”. • Zero and NA do exist but are distinct. Assign “0” if work does not meet benchmark (cell one) performance level. Assign “not applicable” if the student work is not intended to meet a particular criterion.

9. How Have Campuses Used Rubrics to Improve Learning? • Using the VALUE Rubrics for Improvement of Learning and Authentic Assessment • 12 Case Studies • Frequently asked questions http://www.aacu.org/value/casestudies/

10. When presenting assessment results…

11. Consider the Type of Raw Data • Rubrics describe categories. • These categories are generally ordered. (4 reflects higher quality performance than 3.) • But they are likely not equally distributed on a number line. ( 1 is not the same distance from 2 as 2 is from 3. 4 is not twice a “good” as 2.) • Therefore, summary presentations and statistical procedures should be appropriate for ordinal numbers.

12. Consider the Readers • Disciplines have commonly used styles of presenting data. • Individuals have personal learning styles. • Consider presenting the data in multiple forms that will resonate with different styles. • Charts • Graphs • Verbally, qualitatively (rubrics provide quality descriptions for each level)

13. Chart The percent of students meeting the adopted benchmark for lower-division courses.

14. Example Combination Chart and Graph

15. Example Verbal Description CT4 Student’s Position Less than one in twenty work products contained no statement of student position (scores of 0). One fourth of the work products provided a simplistic or obvious position (scores of 1). Three in ten work products provided a specific position that acknowledged different sides of an issue (scores of 2). One third of the work products took into account the complexities of the issue and acknowledged the points of view of others (scores of 3). And less than one in ten work products provided an imaginative position that took into account the complexities of the issue, synthesized others’ viewpoints into the position, and acknowledged the limits of the position taken(scores of 4)

16. Scenario You will work in table groups to analyze and then discuss recent assessment findings and how they might be used to improve student learning.

17. Share Out