Using rubrics to assess student learning
Download
1 / 51

Using Rubrics to Assess Student Learning - PowerPoint PPT Presentation


  • 143 Views
  • Uploaded on

Using Rubrics to Assess Student Learning. Emily Langdon, PhD Coordinator for Assessment, Research, and Evaluation Division of Student Affairs Laura Martin, PhD Coordinator for Institutional Assessment Office of Institutional Assessment Anne Zanzucchi , PhD

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Using Rubrics to Assess Student Learning' - gefjun


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Using rubrics to assess student learning

Using Rubrics to Assess Student Learning

Emily Langdon, PhD

Coordinator for Assessment, Research, and Evaluation

Division of Student Affairs

Laura Martin, PhD

Coordinator for Institutional Assessment

Office of Institutional Assessment

Anne Zanzucchi, PhD

Faculty Development in Assessment Coordinator

Center for Research on Teaching Excellence


Learning outcomes for today
Learning Outcomes for Today:

  • Participant will be…

  • Able to describe what a rubric is and list benefits of using rubrics

  • Familiar with different kinds of rubrics

  • Able to describe the role rubrics play in assessment that provides information for program planning

  • Able to use a rubric to assess student work to gain actionable information on student learning


What is a learning outcome
What is a Learning Outcome?

  • A change in attitude, aptitude or behavior that a student can describe or demonstrate after participating in a program(s) or using a service(s) (CSU Sacramento)

  • Ex. As a result of the health awareness workshop, students can explain how exercise affects stress.

  • Ex. As a result of the time management workshop, students have identified two tools to better manage their schedule.


What is a learning outcome1
What is a Learning Outcome?

  • Term refers both to intended and actual, observed outcomes

  • In doing so, it clarifies the role of LOs in program planning


Role of los in program planning
Role of LOs in program planning

Intended LO

Actual LO


This is the assessment cycle
This is the “assessment cycle”

Step 2 in Assessment Cycle: Conclusions

Step 1 in SA Assessment Cycle: Identify LOs, POs & measures

Step 2 in SA Assessment Cycle: Results


At what levels are los developed assessed
At what levels are LOs Developed & assessed?

Institution

Division of Student Affairs

Academic

SA Department

SA Department

Program

Program

Program

Program

Program

Program

Program

Program


What is a rubric? what roles do they play in the assessment of student learning and program planning in student affairs?


What is a rubric
What is a rubric? of student learning and program planning in student affairs?

  • A scoring guide:

    • a list or chart that articulates the criteria and standards of achievement to be used to evaluate work (Suskie, 2009)

  • a set of criteria specifying the characteristics of an outcome and the levels of achievement for each characteristic (J. Levy, 2012)


What are rubrics used to score
What are rubrics used to score? of student learning and program planning in student affairs?

  • Assessments that require the observation of a performance or behavior(s), ex.

  • Presentation

  • Role plays

  • Teamwork

  • Performances

  • Assessments of written or visual artifacts, ex.

  • Reflection papers

  • Journals

  • Resumes

  • Portfolios

  • Art pieces


What is the relationship between learning outcome and a rubric
What is the relationship between learning outcome and a rubric?

  • Learning Outcome describes what students will do to demonstrate their learning, ex.

  • Give a presentation

  • Role play a….

  • Work in team to…

  • Write a reflection that…

  • Rubric describes

    • The expected properties of that demonstration(criteria)

    • The possible levels of achievement/performance (standards)


Example learning outcome and rubric pairing
Example learning outcome and Rubric pairing rubric?

Learning Outcome:

Reflecting teamwork instruction in the leadership workshop, students will employ their teamwork skills in the execution of their final group project.

  • Scoring Rubric:

  • Teamwork rubric defines four criteria and related standards for effectiveteamwork that will be examined through the interactions needed to complete the final group project:

    • Contributing to team meetings

    • Facilitating the contributions of others

    • Fostering constructive team climate

    • Responding to conflict


Types of rubrics
Types of rubrics rubric?

  • Rubrics differ in structure - in the way criteria and performance standards are described. And thus,

  • In the kind of information they can provide about the quality of a piece of work

  • Choice of rubric depends upon what you want to use the information for.

  • All rubrics enable collection of qualitative and quantitative data


Major types of rubrics
Major Types of Rubrics rubric?

  • Checklist

  • Rating Scale

  • Descriptive (also called analytic)

  • Holistic





Example of a holistic rubric for assessing student essay allen 2004
Example of a Holistic Rubric For Assessing Student Essay Members (Allen, 2004)

Standards

Criteria


Where do rubrics fit into cycle

Identify learning evidence to be collected and that will use a rubric to score

Where do rubrics fit into cycle?

Might refine rubric as part of this step

Use rubric to score student work (measure learning)

… Develop or identify and pilot rubric here


Activity scoring student work with a rubric
Activity: use a rubric to scoreScoring student work with a rubric.

Steps:

  • Score student work using rubric.

  • Review data for inter-rater reliability. What if differences in scoring?

  • Summarize results – how best to do that?

  • Identify possible actions suggested by results.


Activity scoring student work with a rubric1
Activity: use a rubric to scoreScoring student work with a rubric.

Do set up with learning outcome….. And pull in assessment cycle. Working in pairs, use the rubric to score a set of student work. Provide scores to Laura to enter in spreadsheet.


Review scores check for inter rater reliability
Review Scores: Check for Inter-rater reliability. use a rubric to score

  • Examine degree to which scores for same paper agree.

  • If diverge, why?

  • What do?

    • Ask another rater to score

    • Revise rubric to clarify

    • Best practice: “Calibrate” reviewers to apply rubric before scoring using example work so that reviewers share understanding of how to apply rubric


Summarize results
Summarize Results use a rubric to score

  • Goal is information on which we can act, so frequencies with which particular levels of performance are observed are preferred relative to average performance for the group.

    • Do our example – calculate a mean and represent the results as a frequency, which is more intuitive to folks?


Summarize results other problems with averages
Summarize Results: Other problems with averages use a rubric to score

  • Averages may not appropriately describe distribution of student performance. Ex.

  • ADD an example here using our rubric….showing that a mean may show that student performance is just fine but really it is dichotomous, students either did poorly or very well.

  • Averages are not appropriate because they assume that the difference between each standard is equal.


Interpreting results and taking action what do our results suggest to you
Interpreting results and taking action: What do our results suggest to you?

1) Consider student learning results:

  • Are you satisfied? If not, what might you do?

    2) Consider usefulness of rubric:

  • Does it work well? Could it be improved? How?


Resources for rubrics
Resources for rubrics suggest to you?

  • Google – tons out there

  • Professional societies, colleagues, networks, listservs

  • VALUE Rubrics

  • Rubistar - http://rubistar.4teachers.org/

  • Develop it yourself


What makes a useful rubric
What makes a useful rubric? suggest to you?

  • The assignment asks students to produce work that addresses the rubric criteria (assignment, instruction, and rubric criteria are aligned).

  • Design rubric before giving the assignment, not after.

  • As possible, give rubric to students with assignment. Consider having them self-score before turning assignment in.

  • Align rubric criteria with learning outcome and the assignment that asks students to produce the outcome.


What makes a useful rubric1
What makes a useful rubric? suggest to you?

  • Criteria and standards are sufficiently distinct and clear that raters consistently apply them in the same way (inter-rater reliability)

  • Use an even number of standards (ex. 4) to avoid tendency to score in middle.

  • The rubric is designed to yield the kind of information about student abilities needed to make planning decisions.


How can refine our rubric before we use it to conduct assessment
How can refine our rubric before we use it to conduct assessment?

  • Develop or identify rubric shaping it to needs

  • Pilot rubric by applying it to example work

  • Share rubric with students to understand how they interpret it

  • Revise/refine rubric to increase inter-rater reliability and usefulness to students


Activity what might you use a rubric for in your own work
Activity: assessment?What might you use a rubric for in your own work?

Reflect on your own program work,

  • Identify an opportunity to apply a rubric.

  • What would you use the rubric to evaluate? Why?

  • What kinds of rubric (analytic, holistic, etc.) would you choose and why?


What are some advantages of using rubrics to students suskie 2009
What are some advantages of using rubrics to students? assessment?(suskie, 2009)

  • Provide to students to help them understand your expectations. Best practice – provide with the assignment

  • Students can apply rubrics themselves to guide learning and improvement (ex. self-score before turn in)

  • Improves feedback to students by identifying specific areas for improvement

  • Gather information that can be used to improve instruction by highlighting challenges shared by a significant proportion of students


What are some advantages of using rubrics for teachers and programs suskie 2009
What are some advantages of using rubrics for teachers and programs?(suskie, 2009)

  • Development of criteria and standards help to clarify vague or fuzzy goals for a program and among colleagues

  • Makes expectations public and shared; all colleagues can understand precisely what intend students to be able to do

  • Make scoring easier and faster

  • Increase consistency in scoring across students, among raters, and through time; allow to measure improvement

  • Can gather complementary indirect evidence through student self-ratings

  • Makes it easier to identify common strengths and weaknesses in student work, behavior, etc. to inform program planning.


What makes a good learning outcome
What makes a Good Learning Outcome? programs?

  • Written from perspective “Students will be able to….”

  • Includes an action verb describing specifically what students will be able to do if they have learned

  • Describes how students will demonstrate their learning (i.e. is measurable)

  • Yield information that can be used to refine programming (i.e. actionable information)

  • Is understandable to students!


Example learning outcomes
Example Learning Outcomes programs?

Original: Participants will be able to appreciate the benefits of exercise

Revised: Participants will be able to explain how exercise affects stress


Example learning outcomes1
Example Learning Outcomes programs?

Original: Participants will have more confidence in their abilities to address racial discrimination

Revised: Through role play, participants will demonstrate their ability to analyze and respond to arguments about racial discrimination


Activity refining los
Activity: programs?Refining LOs

Original: Participants will understand the nine reasons for conducting a needs assessment.

  • With a partner, answer the following questions and, as necessary, develop a revised statement:

  • Is the verb active, specifying what students can do if they have learned?

  • Can it be measured? Does it describe how students will demonstrate their learning?

  • Will it yield actionable information?

  • Is it understandable to students?


Activity refining los1
Activity: programs?Refining LOs

Original: As a result of the time management workshop, students will arrive to class on time daily.

  • With a partner, answer the following questions and, as necessary, develop a revised statement:

  • Is the verb active, specifying what students can do if they have learned?

  • Can it be measured? Does it describe how students will demonstrate their learning?

  • Will it yield actionable information?

  • Is it understandable to students?


What kind of learning evidence is often most easily collected and actionable
What kind of learning evidence is often most easily collected and “actionable”?

Direct Evidence:

  • Evidence of learning that is tangible, visible; actual student work or demonstrations of student ability.

    Exs.

    • Written or oral explanations, descriptions, identifications, etc.

    • Role plays

    • Lists

    • Project products

    • Etc.


What is indirect evidence
What is indirect evidence? collected and “actionable”?

Indirect Evidence:

  • Proxy signs that students are learning, for example, student self-perceptions of abilities.

    Exs.

    • Questions that ask students to rate their own abilities

    • Questions that ask students to rate gains in abilities or knowledge (ex. Relative to the start of this workshop, how well can you…)

    • Satisfaction type questions on surveys


What is the relationship of los to measures in the sa template
What is the relationship of LOs to measures in the SA template?

  • A refined LO will clarify the methodology for assessment because it indicates how students will demonstrate their learning. Examples….

    • “…explain how exercise affects stress.”

    • “…identify the most appropriate university resource for addressing their academic or personal concern or need.”

    • “…list three reasons for…”


Worksheet activity developing and revising your own outcomes
Worksheet activity: template?Developing and revising your own outcomes


Benefits of learning outcomes for the teacher and department
Benefits of Learning Outcomes… for the Teacher and Department

  • Framework for program design/development = content and activities

  • Reference point for “researching” student learning, yielding insights relevant to future offerings

  • Articulates intentions of programming/service, which provides opportunities for educators to

    • Discuss and improve educational offerings

    • Align educational offerings within and among departments (co-curricular and academic) and in relation to division and institutional goals for student learning


Benefits of learning outcomes for the teacher and department1
Benefits of Learning Outcomes… for the Teacher and Department

  • Advertise value and impact of programming to build interest and buy-in of students, faculty, student affairs colleagues at UC Merced and beyond, alumni and potentially even donors!

  • Builds integrated picture of programming within department to support planning…

    • What are we doing collectively? Is it what we intend?

    • Are we investing our most precious resource (time) in ways that maximize our intended educational impact?


Benefits of learning outcomes for students
Benefits of Learning Outcomes… for Students Department

  • Explains to students why they should engage in programming or service

  • Supports student stewardship of their learning

    • Choosing to participate in light of personal goals and interests

    • Monitoring learning during program

  • Supports holistic integrated view of education, ex. connect classroom and co-curricular learning

  • Helps students to see that all experiences have educational value, enabling development as life long learners (an SA Divisional Outcome)


Activity: DepartmentWith a partner, compare and contract the example types* of rubrics & address the following questions.

Imagine using the rubric to score student work:

  • What do you like about each type of rubric? What don’t you like? Why?

  • What if any rubric type(s) would you prefer to use or not to use? Why?

  • As you compare and contrast each type of rubric record your observations.

* Please focus on the rubric’s structure and your thoughts about its usefulness, not the actual criteria and standards.


Where do rubrics fit into cycle1
Where do rubrics fit into cycle? Department

  • A rubric is a scoring guide

  • A tool for determining to what degree students have met the learning outcome (measuring learning)


Example of hierarchical lo alignment
Example of hierarchical LO alignment Department

Institution: Students will understand and value diverse perspectives in both the global and community contexts of modern society in order to work knowledgeably and effectively in an ethnically and culturally rich setting (Self and Society)

Program – Cultural Expression: 100% of participants in the cultural expression workshop will identify an example of cultural expression they have observed and describe a contribution it makes to the UC Merced community.

Division: Develop an understanding and appreciation of human differences

Department – Housing and Residential Life: Through informal and formal residential life experiences and programming, resident interest in and respect for new ideas and cultural and life style differences increases.


Activity at what level are you developing assessing outcomes
Activity: At what level are you developing & assessing outcomes?

  • Review previous assessment work, at what level was student learning assessed?


Where do los come from
Where do LOs come from? outcomes?

  • Derived from/support departments goals

  • Address worthwhile and shared learning priorities

  • LOs are specific examples of how a department goal is met.


An example 3 l o s that meet same departmental goal
An Example: 3 L.O.s that meet same departmental goal outcomes?

Housing and Residential Life Goal: Housing and Residence Life is committed to providing residential life experiences and programming that foster resident interest in and respect for new ideas and cultural and life style differences.

  • Cultural Expression Program LO: 100% of participants in the cultural expression workshop will identify an example of cultural expression they have observed and describe a contribution it makes to the UC Merced community.

  • Values Clarification Program LO: 100% of participants in the Values Clarification Workshop will be able to describe three benefits to valuing and respecting cultural differences

  • Peer Pressure Program LO: Given a typical scenario, 100% of participants in the Peer Pressure Workshop will identify two strategies they can use to support and promote acceptance and tolerance of difference.


ad