Criterion based vs norm based evaluation
Download
1 / 29

Criterion Based vs. Norm-Based Evaluation - PowerPoint PPT Presentation


  • 268 Views
  • Updated On :

“Criterion Based” vs. “Norm-Based” Evaluation. David L Gaspar MD October 18, 2008. Evaluation of Students. Goals- This session is designed to enable participants to: 1. Understand the definition and purpose of evaluation 2. Develop a framework to make evaluation efficient and useful

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Criterion Based vs. Norm-Based Evaluation' - awen


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Criterion based vs norm based evaluation l.jpg

“Criterion Based” vs. “Norm-Based” Evaluation

David L Gaspar MD

October 18, 2008

UCD School of Medicine


Evaluation of students l.jpg
Evaluation of Students

Goals-This session is designed to enable participants to:

1. Understand the definition and purpose of evaluation

2. Develop a framework to make evaluation efficient and useful

3. Understand and apply criterion-based evaluation – the “R.I.M.E.” method

UCD School of Medicine


Evaluation basics l.jpg
Evaluation Basics

Definition

Evaluation is the process of making judgments based on factual information and observations in order to rate, rank, or assess an individual’s status at a given point.

Purposes of Evaluation

1. Summarize performance at a given point in time.

2. Provide information for planning future educational experiences.

3. Communicate summary information to other parties.

UCD School of Medicine


Framework for planning clerkship evaluation l.jpg
Framework for Planning Clerkship Evaluation

Before Clerkship

1. Understand medical school expectations and “forms”.

2. Review clerkship goals and objectives.

During Clerkship

1. Gather information from multiple sources & provide feedback.

3. Use systematic method of recording.

End of Clerkship

1. Prepare for final evaluation.

2. Schedule and conduct summary meeting.

3. Complete and submit final evaluation.

UCD School of Medicine


Tips to facilitate evaluation l.jpg
Tips to Facilitate Evaluation

  • Dictate or write a student progress note. (end of day, when procedures occur-behaviours)

  • Use student log books.

  • Review student’s written record. (copy progress notes at the beginning, middle, and end of clerkship)

  • Use student self-assessment forms.

  • Use computer printout of student’s patient profile.

UCD School of Medicine


Evaluate l.jpg
Evaluate…

End of Clerkship

1. Review evaluation criteria and form.

2. Review student performance data.

  • Schedule and conduct final evaluation session with student.

  • Have some “backbone”

    5. Complete and return the evaluation form promptly. (Keep a Copy!!!)

UCD School of Medicine


Norm based vs criterion based l.jpg
“Norm”-Based vs. Criterion-Based

Any ideas????

UCD School of Medicine


Norm based vs criterion based8 l.jpg
“Norm”-Based vs. Criterion-Based

NORM-BASED EVALUATION

  • Norm-based assessment compares individuals’ performances to one another, resulting in a ranking of individuals.

    Medical Knowledge

UCD School of Medicine


Norm based vs criterion based9 l.jpg
“Norm”-Based vs. Criterion-Based

NORM-BASED EVALUATION

  • This traditional form of assessment has been widely used to select individuals for further education or for jobs and is what most students, residents, fellows and faculty have been exposed to throughout their training

UCD School of Medicine


Norm based vs criterion based10 l.jpg
“Norm”-Based vs. Criterion-Based

NORM-BASED EVALUATION

  • Since norm-based assessment provides no information about the observed quality of the performance, it isn’t useful for providing feedback or improving performance.

  • The rater’s task ends up distinguishing one individual’s performance from another’s.

UCD School of Medicine


Norm based vs criterion based11 l.jpg
“Norm”-Based vs. Criterion-Based

NORM-BASED EVALUATION

  • Raters will vary in their assessments depending on their experience with using a norm-based scale, and their experience with the group being evaluated

UCD School of Medicine


Slide12 l.jpg

Dr. Thomas, who has 15 years of experience working with and evaluating students, is due to evaluate John, his student over the last 2 weeks. He has observed John performing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. Dr. Thomas has a clear expectation based on this experience for what an “honors student” is. He believes that John is a very good student with great enthusiasm, but not quite functioning at an honors level and will therefore complete his evaluation in a manner that reflects what he thinks will result in a “high pass” grade- on a 6-point scale. mostly 5s and some 6s.

1= poor performance and 6=excellent performance,

NORM-BASED EVALUATION

UCD School of Medicine


Slide13 l.jpg

Dr. Thomas’ evaluation: evaluating students, is due to evaluate John, his student over the last 2 weeks. He has observed John performing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. Dr. Thomas has a clear expectation based on this experience for what an “honors student” is. He believes that John is a very good student with great enthusiasm, but not quite functioning at an honors level and will therefore complete his evaluation in a manner that reflects what he thinks will result in a “high pass” grade- on a 6-point scale. mostly 5s and some 6s.

1= poor performance and 6=excellent performance,

NORM-BASED EVALUATION

UCD School of Medicine


Slide14 l.jpg

The next attending to work with John, Dr. Smith, has just completed residency. She too has been working with John for 2 weeks and has observed him doing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. She is an enthusiastic teacher and believes that students should get honors if they are active participants in rounds, enthusiastic about learning and generally helpful. She went to a medical school where most students got honors if they did these things and she thinks this is a good policy. She has heard from the medical students and residents that she has worked with that to get honors, she must give John all 6s on a 6 -point scale

1= poor performance and 6=excellent performance,

NORM-BASED EVALUATION

UCD School of Medicine


Slide15 l.jpg

Dr. Thomas’ evaluation: completed residency. She too has been working with John for 2 weeks and has observed him doing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. She is an enthusiastic teacher and believes that students should get honors if they are active participants in rounds, enthusiastic about learning and generally helpful. She went to a medical school where most students got honors if they did these things and she thinks this is a good policy. She has heard from the medical students and residents that she has worked with that to get honors, she must give John all 6s on a 6 -point scale

1= poor performance and 6=excellent performance,

NORM-BASED EVALUATION

UCD School of Medicine


Slide16 l.jpg

What grade does John deserve? completed residency. She too has been working with John for 2 weeks and has observed him doing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. She is an enthusiastic teacher and believes that students should get honors if they are active participants in rounds, enthusiastic about learning and generally helpful. She went to a medical school where most students got honors if they did these things and she thinks this is a good policy. She has heard from the medical students and residents that she has worked with that to get honors, she must give John all 6s on a 6 -point scale

More importantly, what is John’s level of competence?

NORM-BASED EVALUATION

UCD School of Medicine


Slide17 l.jpg

Answer: completed residency. She too has been working with John for 2 weeks and has observed him doing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. She is an enthusiastic teacher and believes that students should get honors if they are active participants in rounds, enthusiastic about learning and generally helpful. She went to a medical school where most students got honors if they did these things and she thinks this is a good policy. She has heard from the medical students and residents that she has worked with that to get honors, she must give John all 6s on a 6 -point scale

Hard to tell…..This example provides insight into some of the challenges of norm -based assessment. Now let’s learn about an alternative means of assessment.

NORM-BASED EVALUATION

UCD School of Medicine


How do we evaluate the learner with a criterion based system l.jpg
How do we evaluate the learner with a criterion-based system?

  • There have been a variety of methods used over the past several decades

  • In 1999 L. Pangaro developed RIME – a new tool for evaluating the level of your learner – more specifically for medical students

UCD School of Medicine


Slide19 l.jpg
RIME system?

  • Reporter

  • Interpreter

  • Manager

  • Educator

  • Before RIME – Observer/Pre-Reporter (pre-clinical)

UCD School of Medicine


Slide20 l.jpg
RIME system?

  • Reporter - Consistently good in interpersonal skills, reliably obtains and communicates clinical findings. Written and spoken presentations are clear and organized.

  • Interpreter – Able to prioritize and analyze patient problems, develop a differential diagnosis and next steps in workup. Data is not just data (reporter), but has a clinical significance.

UCD School of Medicine


Slide21 l.jpg
RIME system?

  • Manager – Consistently proposes reasonable diagnostic and therapeutic options, incorporating patient preferences. They can take their knowledge, integrate it with the findings for a given patient and form a plan that prioritizes within and amongst the problem list.

  • Educator – Consistent level of knowledge of current medical evidence; demonstrates self-directed learning (practice based improvement and learning) and contributes to others’ education.

UCD School of Medicine


Rime changes with time l.jpg
RIME Changes with Time system?

  • You can use RIME to track the progress/growth of a student over the course of their clinical years or within a given year.

  • Early MS3 – Should report well and begin to interpret.

  • Late MS3 – Should be a good interpreter

  • Sub-I should have the above and begin to manage patients appropriately.

  • Interns should become astute managers

  • R2s and beyond – progressive depth of management and becoming educators

UCD School of Medicine


Rime criterion based evaluation l.jpg
RIME: Criterion-based evaluation system?

  • Criterion -based assessment begins with an established set of criteria for successful performance and compares an individual’s performance to these criteria.

UCD School of Medicine


Rime criterion based evaluation24 l.jpg
RIME: Criterion-based evaluation system?

  • Rating an individual’s performance against the criteria provides feedback on the quality of performance. Multiple individuals may successfully meet the criteria as one individual’s performance is not constrained by another’s performance. Agreement between raters is high as the standard for comparison is stable and based on observable behaviors. In other words, the rater is responsible for documenting what the specific skills of the individual are based on their observed performance. They are NOT responsible for comparing them to anyone else, or for trying to determine a grade.

UCD School of Medicine


Rime criterion based evaluation25 l.jpg
RIME: Criterion-based evaluation system?

In the scenario just described, both Dr. Smith and Dr. Thomas have observed John’s oral presentations. When they are asked to complete the evaluation of John they are asked to rate his abilities based on observable behaviors related to his oral presentations. John’s oral presentations are accurate and objective. But, they sometimes contain extraneous information and important information gets lost in the presentation.

Please complete the evaluation below as if you were Dr. Thomas, with 15 years of experience teaching medical student:

UCD School of Medicine


Rime criterion based evaluation26 l.jpg
RIME: Criterion-based evaluation system?

Would you evaluate the student any differently if you were Dr. Smith?

UCD School of Medicine


Rime criterion based evaluation27 l.jpg
RIME: Criterion-based evaluation system?

John is able to derive a rudimentary differential diagnosis and assessment for patients presenting with shortness of breath and chest pain with only minor assistance from an intern or resident. But, he has no idea how to approach the presenting problems of acute renal failure and hyponatremia. For each of these latter clinical situations, the intern needed to completely develop the differential diagnosis and teach it to John, who was then able to repeat it at attending rounds. He is only able to develop management plans with significant help from the intern. Each of these 4 conditions is common on your rotation and required clinical entities for the clerkship.

How would you rate him on the following scale?

UCD School of Medicine


Rime criterion based evaluation28 l.jpg
RIME: Criterion-based evaluation system?

John is able to derive a rudimentary differential diagnosis and assessment for patients presenting with shortness of breath and chest pain with only minor assistance from an intern or resident. But, he has no idea how to approach the presenting problems of acute renal failure and hyponatremia. For each of these latter clinical situations, the intern needed to completely develop the differential diagnosis and teach it to John, who was then able to repeat it at attending rounds. He is only able to develop management plans with significant help from the intern. Each of these 4 conditions is common on your rotation and required clinical entities for the clerkship.

How would you rate him on the following scale?

UCD School of Medicine


And then its in my court l.jpg
And Then Its in My Court… system?

UCD School of Medicine


ad