Using rubrics to assess student learning
This presentation is the property of its rightful owner.
Sponsored Links
1 / 51

Using Rubrics to Assess Student Learning PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on
  • Presentation posted in: General

Using Rubrics to Assess Student Learning. Emily Langdon, PhD Coordinator for Assessment, Research, and Evaluation Division of Student Affairs Laura Martin, PhD Coordinator for Institutional Assessment Office of Institutional Assessment Anne Zanzucchi , PhD

Download Presentation

Using Rubrics to Assess Student Learning

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Using rubrics to assess student learning

Using Rubrics to Assess Student Learning

Emily Langdon, PhD

Coordinator for Assessment, Research, and Evaluation

Division of Student Affairs

Laura Martin, PhD

Coordinator for Institutional Assessment

Office of Institutional Assessment

Anne Zanzucchi, PhD

Faculty Development in Assessment Coordinator

Center for Research on Teaching Excellence


Learning outcomes for today

Learning Outcomes for Today:

  • Participant will be…

  • Able to describe what a rubric is and list benefits of using rubrics

  • Familiar with different kinds of rubrics

  • Able to describe the role rubrics play in assessment that provides information for program planning

  • Able to use a rubric to assess student work to gain actionable information on student learning


What is a learning outcome

What is a Learning Outcome?

  • A change in attitude, aptitude or behavior that a student can describe or demonstrate after participating in a program(s) or using a service(s) (CSU Sacramento)

  • Ex. As a result of the health awareness workshop, students can explain how exercise affects stress.

  • Ex. As a result of the time management workshop, students have identified two tools to better manage their schedule.


What is a learning outcome1

What is a Learning Outcome?

  • Term refers both to intended and actual, observed outcomes

  • In doing so, it clarifies the role of LOs in program planning


Role of los in program planning

Role of LOs in program planning

Intended LO

Actual LO


This is the assessment cycle

This is the “assessment cycle”

Step 2 in Assessment Cycle: Conclusions

Step 1 in SA Assessment Cycle: Identify LOs, POs & measures

Step 2 in SA Assessment Cycle: Results


At what levels are los developed assessed

At what levels are LOs Developed & assessed?

Institution

Division of Student Affairs

Academic

SA Department

SA Department

Program

Program

Program

Program

Program

Program

Program

Program


Using rubrics to assess student learning

What is a rubric? what roles do they play in the assessment of student learning and program planning in student affairs?


What is a rubric

What is a rubric?

  • A scoring guide:

    • a list or chart that articulates the criteria and standards of achievement to be used to evaluate work (Suskie, 2009)

  • a set of criteria specifying the characteristics of an outcome and the levels of achievement for each characteristic (J. Levy, 2012)


What are rubrics used to score

What are rubrics used to score?

  • Assessments that require the observation of a performance or behavior(s), ex.

  • Presentation

  • Role plays

  • Teamwork

  • Performances

  • Assessments of written or visual artifacts, ex.

  • Reflection papers

  • Journals

  • Resumes

  • Portfolios

  • Art pieces


What is the relationship between learning outcome and a rubric

What is the relationship between learning outcome and a rubric?

  • Learning Outcome describes what students will do to demonstrate their learning, ex.

  • Give a presentation

  • Role play a….

  • Work in team to…

  • Write a reflection that…

  • Rubric describes

    • The expected properties of that demonstration(criteria)

    • The possible levels of achievement/performance (standards)


Example learning outcome and rubric pairing

Example learning outcome and Rubric pairing

Learning Outcome:

Reflecting teamwork instruction in the leadership workshop, students will employ their teamwork skills in the execution of their final group project.

  • Scoring Rubric:

  • Teamwork rubric defines four criteria and related standards for effectiveteamwork that will be examined through the interactions needed to complete the final group project:

    • Contributing to team meetings

    • Facilitating the contributions of others

    • Fostering constructive team climate

    • Responding to conflict


Types of rubrics

Types of rubrics

  • Rubrics differ in structure - in the way criteria and performance standards are described. And thus,

  • In the kind of information they can provide about the quality of a piece of work

  • Choice of rubric depends upon what you want to use the information for.

  • All rubrics enable collection of qualitative and quantitative data


Major types of rubrics

Major Types of Rubrics

  • Checklist

  • Rating Scale

  • Descriptive (also called analytic)

  • Holistic


Example of a checklist rubric for a website suskie 2009

Example of a Checklist rubric for a website (suskie, 2009)


Example of a rating scale rubric for evaluating fellow group members suskie 2009

Example of a Rating Scale Rubric for Evaluating Fellow Group Members (suskie, 2009)

Standards


Example of a descriptive analytic rubric

Example of a Descriptive (Analytic) rubric

Standards

Criteria


Example of a holistic rubric for assessing student essay allen 2004

Example of a Holistic Rubric For Assessing Student Essay (Allen, 2004)

Standards

Criteria


Where do rubrics fit into cycle

Identify learning evidence to be collected and that will use a rubric to score

Where do rubrics fit into cycle?

Might refine rubric as part of this step

Use rubric to score student work (measure learning)

… Develop or identify and pilot rubric here


Activity scoring student work with a rubric

Activity: Scoring student work with a rubric.

Steps:

  • Score student work using rubric.

  • Review data for inter-rater reliability. What if differences in scoring?

  • Summarize results – how best to do that?

  • Identify possible actions suggested by results.


Activity scoring student work with a rubric1

Activity: Scoring student work with a rubric.

Do set up with learning outcome….. And pull in assessment cycle. Working in pairs, use the rubric to score a set of student work. Provide scores to Laura to enter in spreadsheet.


Review scores check for inter rater reliability

Review Scores: Check for Inter-rater reliability.

  • Examine degree to which scores for same paper agree.

  • If diverge, why?

  • What do?

    • Ask another rater to score

    • Revise rubric to clarify

    • Best practice: “Calibrate” reviewers to apply rubric before scoring using example work so that reviewers share understanding of how to apply rubric


Summarize results

Summarize Results

  • Goal is information on which we can act, so frequencies with which particular levels of performance are observed are preferred relative to average performance for the group.

    • Do our example – calculate a mean and represent the results as a frequency, which is more intuitive to folks?


Summarize results other problems with averages

Summarize Results: Other problems with averages

  • Averages may not appropriately describe distribution of student performance. Ex.

  • ADD an example here using our rubric….showing that a mean may show that student performance is just fine but really it is dichotomous, students either did poorly or very well.

  • Averages are not appropriate because they assume that the difference between each standard is equal.


Interpreting results and taking action what do our results suggest to you

Interpreting results and taking action: What do our results suggest to you?

1) Consider student learning results:

  • Are you satisfied? If not, what might you do?

    2) Consider usefulness of rubric:

  • Does it work well? Could it be improved? How?


Resources for rubrics

Resources for rubrics

  • Google – tons out there

  • Professional societies, colleagues, networks, listservs

  • VALUE Rubrics

  • Rubistar - http://rubistar.4teachers.org/

  • Develop it yourself


What makes a useful rubric

What makes a useful rubric?

  • The assignment asks students to produce work that addresses the rubric criteria (assignment, instruction, and rubric criteria are aligned).

  • Design rubric before giving the assignment, not after.

  • As possible, give rubric to students with assignment. Consider having them self-score before turning assignment in.

  • Align rubric criteria with learning outcome and the assignment that asks students to produce the outcome.


What makes a useful rubric1

What makes a useful rubric?

  • Criteria and standards are sufficiently distinct and clear that raters consistently apply them in the same way (inter-rater reliability)

  • Use an even number of standards (ex. 4) to avoid tendency to score in middle.

  • The rubric is designed to yield the kind of information about student abilities needed to make planning decisions.


How can refine our rubric before we use it to conduct assessment

How can refine our rubric before we use it to conduct assessment?

  • Develop or identify rubric shaping it to needs

  • Pilot rubric by applying it to example work

  • Share rubric with students to understand how they interpret it

  • Revise/refine rubric to increase inter-rater reliability and usefulness to students


Activity what might you use a rubric for in your own work

Activity: What might you use a rubric for in your own work?

Reflect on your own program work,

  • Identify an opportunity to apply a rubric.

  • What would you use the rubric to evaluate? Why?

  • What kinds of rubric (analytic, holistic, etc.) would you choose and why?


What are some advantages of using rubrics to students suskie 2009

What are some advantages of using rubrics to students?(suskie, 2009)

  • Provide to students to help them understand your expectations. Best practice – provide with the assignment

  • Students can apply rubrics themselves to guide learning and improvement (ex. self-score before turn in)

  • Improves feedback to students by identifying specific areas for improvement

  • Gather information that can be used to improve instruction by highlighting challenges shared by a significant proportion of students


What are some advantages of using rubrics for teachers and programs suskie 2009

What are some advantages of using rubrics for teachers and programs?(suskie, 2009)

  • Development of criteria and standards help to clarify vague or fuzzy goals for a program and among colleagues

  • Makes expectations public and shared; all colleagues can understand precisely what intend students to be able to do

  • Make scoring easier and faster

  • Increase consistency in scoring across students, among raters, and through time; allow to measure improvement

  • Can gather complementary indirect evidence through student self-ratings

  • Makes it easier to identify common strengths and weaknesses in student work, behavior, etc. to inform program planning.


What makes a good learning outcome

What makes a Good Learning Outcome?

  • Written from perspective “Students will be able to….”

  • Includes an action verb describing specifically what students will be able to do if they have learned

  • Describes how students will demonstrate their learning (i.e. is measurable)

  • Yield information that can be used to refine programming (i.e. actionable information)

  • Is understandable to students!


Example learning outcomes

Example Learning Outcomes

Original: Participants will be able to appreciate the benefits of exercise

Revised: Participants will be able to explain how exercise affects stress


Example learning outcomes1

Example Learning Outcomes

Original: Participants will have more confidence in their abilities to address racial discrimination

Revised: Through role play, participants will demonstrate their ability to analyze and respond to arguments about racial discrimination


Activity refining los

Activity: Refining LOs

Original: Participants will understand the nine reasons for conducting a needs assessment.

  • With a partner, answer the following questions and, as necessary, develop a revised statement:

  • Is the verb active, specifying what students can do if they have learned?

  • Can it be measured? Does it describe how students will demonstrate their learning?

  • Will it yield actionable information?

  • Is it understandable to students?


Activity refining los1

Activity: Refining LOs

Original: As a result of the time management workshop, students will arrive to class on time daily.

  • With a partner, answer the following questions and, as necessary, develop a revised statement:

  • Is the verb active, specifying what students can do if they have learned?

  • Can it be measured? Does it describe how students will demonstrate their learning?

  • Will it yield actionable information?

  • Is it understandable to students?


What kind of learning evidence is often most easily collected and actionable

What kind of learning evidence is often most easily collected and “actionable”?

Direct Evidence:

  • Evidence of learning that is tangible, visible; actual student work or demonstrations of student ability.

    Exs.

    • Written or oral explanations, descriptions, identifications, etc.

    • Role plays

    • Lists

    • Project products

    • Etc.


What is indirect evidence

What is indirect evidence?

Indirect Evidence:

  • Proxy signs that students are learning, for example, student self-perceptions of abilities.

    Exs.

    • Questions that ask students to rate their own abilities

    • Questions that ask students to rate gains in abilities or knowledge (ex. Relative to the start of this workshop, how well can you…)

    • Satisfaction type questions on surveys


What is the relationship of los to measures in the sa template

What is the relationship of LOs to measures in the SA template?

  • A refined LO will clarify the methodology for assessment because it indicates how students will demonstrate their learning. Examples….

    • “…explain how exercise affects stress.”

    • “…identify the most appropriate university resource for addressing their academic or personal concern or need.”

    • “…list three reasons for…”


Worksheet activity developing and revising your own outcomes

Worksheet activity: Developing and revising your own outcomes


Benefits of learning outcomes for the teacher and department

Benefits of Learning Outcomes… for the Teacher and Department

  • Framework for program design/development = content and activities

  • Reference point for “researching” student learning, yielding insights relevant to future offerings

  • Articulates intentions of programming/service, which provides opportunities for educators to

    • Discuss and improve educational offerings

    • Align educational offerings within and among departments (co-curricular and academic) and in relation to division and institutional goals for student learning


Benefits of learning outcomes for the teacher and department1

Benefits of Learning Outcomes… for the Teacher and Department

  • Advertise value and impact of programming to build interest and buy-in of students, faculty, student affairs colleagues at UC Merced and beyond, alumni and potentially even donors!

  • Builds integrated picture of programming within department to support planning…

    • What are we doing collectively? Is it what we intend?

    • Are we investing our most precious resource (time) in ways that maximize our intended educational impact?


Benefits of learning outcomes for students

Benefits of Learning Outcomes… for Students

  • Explains to students why they should engage in programming or service

  • Supports student stewardship of their learning

    • Choosing to participate in light of personal goals and interests

    • Monitoring learning during program

  • Supports holistic integrated view of education, ex. connect classroom and co-curricular learning

  • Helps students to see that all experiences have educational value, enabling development as life long learners (an SA Divisional Outcome)


Using rubrics to assess student learning

Activity: With a partner, compare and contract the example types* of rubrics & address the following questions.

Imagine using the rubric to score student work:

  • What do you like about each type of rubric? What don’t you like? Why?

  • What if any rubric type(s) would you prefer to use or not to use? Why?

  • As you compare and contrast each type of rubric record your observations.

* Please focus on the rubric’s structure and your thoughts about its usefulness, not the actual criteria and standards.


Where do rubrics fit into cycle1

Where do rubrics fit into cycle?

  • A rubric is a scoring guide

  • A tool for determining to what degree students have met the learning outcome (measuring learning)


Example of hierarchical lo alignment

Example of hierarchical LO alignment

Institution: Students will understand and value diverse perspectives in both the global and community contexts of modern society in order to work knowledgeably and effectively in an ethnically and culturally rich setting (Self and Society)

Program – Cultural Expression: 100% of participants in the cultural expression workshop will identify an example of cultural expression they have observed and describe a contribution it makes to the UC Merced community.

Division: Develop an understanding and appreciation of human differences

Department – Housing and Residential Life: Through informal and formal residential life experiences and programming, resident interest in and respect for new ideas and cultural and life style differences increases.


Activity at what level are you developing assessing outcomes

Activity: At what level are you developing & assessing outcomes?

  • Review previous assessment work, at what level was student learning assessed?


Where do los come from

Where do LOs come from?

  • Derived from/support departments goals

  • Address worthwhile and shared learning priorities

  • LOs are specific examples of how a department goal is met.


An example 3 l o s that meet same departmental goal

An Example: 3 L.O.s that meet same departmental goal

Housing and Residential Life Goal: Housing and Residence Life is committed to providing residential life experiences and programming that foster resident interest in and respect for new ideas and cultural and life style differences.

  • Cultural Expression Program LO: 100% of participants in the cultural expression workshop will identify an example of cultural expression they have observed and describe a contribution it makes to the UC Merced community.

  • Values Clarification Program LO: 100% of participants in the Values Clarification Workshop will be able to describe three benefits to valuing and respecting cultural differences

  • Peer Pressure Program LO: Given a typical scenario, 100% of participants in the Peer Pressure Workshop will identify two strategies they can use to support and promote acceptance and tolerance of difference.


  • Login