Student assessment new ideas and old basics
Download
1 / 76

Student Assessment: New Ideas and Old Basics - PowerPoint PPT Presentation


  • 140 Views
  • Updated On :

26 September 2007. Student Assessment: New Ideas and Old Basics. Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed Service University of the Health Sciences. “ clinical assessment”. By teachers (house staff and faculty)

Related searches for Student Assessment: New Ideas and Old Basics

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Student Assessment: New Ideas and Old Basics' - dea


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Student assessment new ideas and old basics l.jpg

26 September 2007

Student Assessment:New Ideas and Old Basics

Louis Pangaro, MD

Professor and Vice-chair

for Educational Programs

Department of Internal Medicine

Uniformed Service University of the Health Sciences


Clinical assessment l.jpg
clinical assessment”

  • By teachers (house staff and faculty)

  • On clinical rotations

  • Based on words = descriptors

  • Using words = descriptive

  • In vivo (vs. in vitro end of course/year)


What s old l.jpg
What’s old?

  • Suspicion of grades by teachers

  • Teachers reluctance to be direct (honest?)

  • Belief that grading by teachers is subjective

  • Lawsuits about low grades

    • Jamieson, Guidebook for Clerkship Directors, 2005.


What s basic l.jpg
What’s basic?

  • We have an obligation

  • Fairness

  • Mentoring

    • The Hippocratic Oath

  • Professionalism

    • Duty and expertise (Pellegrino)


How it looked in 1987 l.jpg
How it looked in 1987

  • lack of meaningful comments by evaluators

  • insufficient definition of evaluation criteria

  • too much inter-observer variability

  • late submission of evaluations

  • delay in feedback to students

    Tonesk X, Buchanan RG. J Med Ed. 1987


How it looked in 2001 l.jpg
How it looked in 2001

  • “Areas of weakness in current [clinical] evaluation models include psychometric properties associated with the tools, namely their questionable reliability and validity.”

  • Turnbull, International Handbook of Research in Medical Education, 2002.


How it looks in 2007 l.jpg
How it looks in 2007

  • “Constructive criticism is hard to come by..”

  • “…Candor is at least as painful to the provider as to the recipient…”

  • “…faculty members feel uncomfortable inflicting pain even in a good cause like student improvement.”

  • HMS Student Handbook, 2006-2007


Focus l.jpg
focus

  • The emotional issues for teachers and learners in the grading process.

  • Dealing with barriers to candor.


Slide9 l.jpg


What s new l.jpg
What’s new performance.

  • A believe that quality methods can apply to teaching.

  • Search for more rigor.

  • “Best Evididence Med Ed” (BEBM)

  • “Med Ed Research Certificate”

  • MPH, MHPE programs



New focus on evaluation in clincial setting l.jpg
New focus on evaluation performance.in clincial setting:

  • Attention to professional traits (Papadakis)

  • Lawsuits by patients


Context of new methods l.jpg
Context of new methods performance.

  • Objective Structured Clinical Exams

  • 3600 assessments

  • Portfolios

  • Descriptive vocabularies

    • To get more inter-rater agreement

    • To calibrate differences between levels of performance


Focused physical examination l.jpg
FOCUSED PHYSICAL EXAMINATION performance.

  • [Norm]

  • Exam is generally appropriate in scope and technique. Identifies major abnormalities and pertinent normal findings, only occasionally missing elements. Exam linked to history. Appropriate for level of training.

  • HMS Clerkship Grading Form


Focused physical examination 2 l.jpg
FOCUSED PHYSICAL EXAMINATION (2) performance.

  • [Poor]

  • Consistently misses important findings and often does not make appropriate connection between history and physical. Often uses faulty or inappropriate technique. Not organized or thorough.


Focused physical examination 3 l.jpg
FOCUSED PHYSICAL EXAMINATION (3) performance.

  • [Excellent]

  • Exam is consistently superior. Uncovers subtle and important findings, incorporating advanced techniques where appropriate. Exceptionally organized and thorough, even on difficult cases.


American board of internal med l.jpg
American Board of Internal Med performance.

1 2 3 4 5 6 7 8 9

Average and 2 SDs

Minimal acceptable


Acgme outcomes project l.jpg
ACGME Outcomes Project performance.

  • New emphasis on outcomes (vs curriculum)

  • “long-term effort designed to emphasize educational outcome assessment in residency programs and in the accreditation process.”


Acgme outcomes a cultural shift l.jpg
ACGME: Outcomes: a cultural shift performance.

  • Designing Curriculum is no longer enough

  • Results must be demonstrated.

  • A shift from process to product.

  • Content expertise not enough; need pedagogic expertise


Acgme competencies 1999 l.jpg
ACGME “Competencies”(1999) performance.

  • Medical Knowledge

  • Interpersonal & communication skills

  • Professionalism

  • Patient Care

  • System-based Practice

  • Practice-based learning & Improvement



Canmeds 2005 l.jpg

Medical Expert performance.

Communicator

Collaborator

Professional

Manager

Health Advocate

Scholar

CanMEDS (2005)


R i m e scheme l.jpg
“R.I.M.E. Scheme” performance.

  • Reporter

  • Interpreter

  • Manager/Educator

Pangaro, Academic Med, 1999


Acceptance 1 ob gyn l.jpg
Acceptance (1) Ob-Gyn performance.

“The RIME method is a valid, logistically feasible and acceptable way of assessing medical student clinical performance…

…..minimizes disadvantages of descriptive evaluation, and maximizes the opportunity for accurate observations and helpful feedback.”

APGO UME Taskforce:

Espey et al, Am J Ob Gyn, 2007


Slide25 l.jpg

Fairness performance.

Consistency

Expectations


Principle 1 fairness l.jpg
Principle 1 : Fairness performance.

  • to society :

    • valid (not arbitrary) and

    • sensitive to detect marginal performers

  • to students :

    • know what’s expected,

    • timely feedback

  • to teachers :

    • know what observations to make

    • protected (legally, emotionally)


Principle 2 consistency l.jpg
Principle 2 : Consistency performance.

  • Reliability

    • Within teacher

    • Between teachers

      • In same rotation or across blocks

      • Between sites in same clerkship

      • Between disciplines

    • suitable for high-stakes decisions


Strategy l.jpg
strategy performance.

  • Simplicity leads to acceptance and use

  • Acceptance to consistency

  • Consistency to fairness


Fairness consistency l.jpg
Fairness & Consistency performance.

Reliability &Validity =

Stability of measurement and strength of inferences from observations


Old beliefs in our culture l.jpg
“ Old” beliefs in our culture performance.

  • Grading by teachers is subjective, Examinations are objective

  • Measurement > description, Numbers are > words


Framing the question more simply l.jpg
Framing the question more simply: performance.

  • What do we expect of students?

  • Can we get all teachers to have the same expectations, and apply them consistently?

  • [A question of words and of conceptual frameworks]


Slide32 l.jpg

Educational Goals performance.

Curriculum

Evaluation

Feedback / Grading

community

Depends on

expectations


Slide33 l.jpg

The goal: progressive independence of the learner performance.

Learner

Teacher/program

Content – Goals

(Patients)

after SFDP


Frames of reference for expressing goals l.jpg
Frames of reference performance.for expressing goals

  • Analytic

  • Developmental

  • Synthetic


Goals for education generic l.jpg
goals for education (generic): performance.

  • attitudes/behavior

  • skills

  • knowledge

“KSA”


Bloom s taxonomies l.jpg
Bloom’s Taxonomies performance.

Cognitive Domain

Psychomotor Domain

Affective Domain


1 analytic expression of goals l.jpg
1. performance.Analytic expression of Goals

  • “ana-lytic”: takes the learner “apart”

  • into domains, categories

  • “attitude”, “skills”, “knowledge”

  • considered separately

  • generic terms

  • useful for discrete assessments


Curricular goals knowledge l.jpg
Curricular Goals performance.: KNOWLEDGE

The School of Medicine will ensure that before graduation a student will have demonstrated, to the satisfaction of the faculty, the following:

  • The capacity to recognize the limitations in one’s knowledge and clinical skills and to make a commitment to engage in lifelong learning

  • Knowledge of the normal structure and function of each of the major organ systems of the body and the current basic scientific mechanisms operative at the systemic, cellular, and molecular levels

  • Knowledge of the various causes (genetic, developmental, metabolic, toxic, microbiologic, immune, psychosocial, neoplastic, traumatic, and degenerative) of illnesses and diseases

  • Knowledge of the altered structure and function of the body and its major organ systems that are seen in various illnesses and diseases

  • Knowledge of the scientific method in establishing the causation of disease and efficacy of traditional and non-traditional therapies

  • Knowledge of health care policy and the economic, psychological, social, and cultural factors that affect health and health care delivery

  • Knowledge of the most frequent clinical, laboratory, radiographic, and pathologic manifestations of common as well as life threatening diseases

  • Knowledge about relieving pain and ameliorating the suffering of patients

  • Knowledge of the epidemiology of diseases and the systematic approaches useful in promoting health

  • Knowledge of techniques of patient education and counseling in basic lifestyle changes/prevention

  • Knowledge of and approaches to reduce the psychological and physical risks and stresses of the practice of medicine


Curricular goals skills l.jpg
Curricular Goals performance.: SKILLS

  • The School of Medicine will ensure that before graduation a student will have demonstrated, to the satisfaction of the faculty, the following:

  • The ability to obtain an accurate medical history and the ability to perform both a complete and an organ specific examination, including a mental status examination (See appendix A)

  • The ability to perform routine technical procedures (See appendix B)

  • The ability to interpret the results and be aware of the indications, complications, and limitations of commonly used diagnostic procedures (See appendix C)

  • The ability to demonstrate knowledge of theories and principles that govern ethical decision making

  • The ability to reason deductively and inductively in solving clinical problems

  • The ability to construct appropriate differential diagnoses and treatment plans for patients with common conditions, both acute and chronic, including medical, psychiatric, and surgical conditions, and those requiring short- and long-term rehabilitation

  • The ability to recognize patients with immediate life threatening conditions regardless of etiology, and to institute appropriate initial therapy

  • The ability to recognize and outline an initial course of management for patients with serious conditions requiring critical care

  • The ability to communicate effectively, both orally and in writing, with patients, patients’ families, colleagues, and others with whom physicians must exchange information in carrying out their responsibilities

  • The ability to select appropriate tests for detecting patients at risk for specific diseases and to determine strategies for responding appropriately

  • The ability to retrieve, critically review, and effectively utilize biomedical information from electronic databases and other resources for solving problems and making decisions that are relevant to the care of individuals and populations

  • The ability to evaluate the economic, psychosocial, and cultural factors that impact the health of patients and families and to incorporate these into assessment and treatment plan


Curricular goals attitudes l.jpg
Curricular Goals performance.: ATTITUDES

  • The School of Medicine will ensure that before graduation a student will have demonstrated, to the satisfaction of the faculty, the following:

  • A commitment to advocate the interests of one’s patients

  • Compassionate treatment of patients, and respect for their privacy and dignity

  • Honesty and integrity and dutifulness in all interactions with patients, their families, colleagues, and others with whom physicians interact

  • An understanding of, and respect for, the roles of other health care professionals, and the need to collaborate with others in caring for patients and promoting health

  • A commitment to provide care to patients who are unable to pay and to advocate for access to health care for members of underserved populations


2 developmental terms l.jpg

performance.Novice”

“Advanced learner”

“Expert”

2. Developmental Terms:

can identify absence of pulse

can distinguish specific arrythmias

can manage ventricular fibrillation


Dreyfus and dreyfus l.jpg
Dreyfus and Dreyfus performance.

  • Novice

  • Advanced beginner

  • Competent performance

  • Proficient performance

  • Intuitive expert

  • Master

students

residents

facutly

Mind Over Machine(1986)


Slide43 l.jpg

Developmental vs. Analytic performance.

  • time-line, progression included levels of function

  • essential for multi-year training

  • although the terms (“novice”, “master”…) remain generic


Goals acgme competencies l.jpg
Goals: ACGME “Competencies” performance.

  • Medical Knowledge

  • Interpersonal & communication skills

  • Professionalism

  • Patient Care

  • Practice-based learning

  • System-based Practice

What about

the last three??

analytic?

developmental?


Slide45 l.jpg

A cube to encompass competence performance.

UME --> GME  CME

school

Medical Knowledge

Patient Care

Interpersonal skills

Professionalism

Practice-based

learning/improv.

System-based

Practice

hospital

clinic


Resistance is futile l.jpg
Resistance is futile performance.


Slide47 l.jpg

Educational Goals performance.

Curriculum

Evaluation

Feedback / Grading

community

Depends on

expectations


Slide48 l.jpg

3. the “Synthetic”framework performance.

  • “syn-thetic” -putting the learner back together

  • “K S A” are all required, integrated

  • terms are a bit less generic, more behavioral


R i m e scheme49 l.jpg
“R.I.M.E. Scheme” performance.

  • Reporter

  • Interpreter

  • Manager/Educator

Pangaro, Academic Med, 1999


A framework to classify level of function l.jpg
a framework to classify level of function performance.

  • rudimentary reporting: “My patient has a fever, cough and a bad rash - it’s vesicular or pustular ….”

  • rudimentary interpreting: “I think it might be due to chicken pox or herpes.”


Slide51 l.jpg

  • rudimentary manager/educator performance.: “I’d consider a smear of the fluid and a chest x-ray . …We might observe or treat with acyclovir, …but I’m not sure. I’ll have to look this up.”


Alternative model r i m e l.jpg
Alternative model R.I.M.E. performance.

  • Framework that is

    • developmental

    • behavioral

      • student can visualize, framework for observer

    • “synthetic”


Slide53 l.jpg
“RIME” performance.

  • Reporter: takes ownership of getting the facts on every patient (“what”?)

  • Interpreter: takes ownership of thinking and explaining (“why”?)…

  • Manager: takes ownership of planning with patient (“how?)…

  • Educator: takes ownership of developing and sharing expertise…


Matrix transition to higher expectations l.jpg
Matrix: transition to higher expectations performance.

MM

M

M

I R R R P

I R P

I R P

I R P

I = introduced R = repetition P = proficiency

M = mastery in practice


Duty and expertise pellegrino l.jpg
Duty and Expertise (Pellegrino) performance.

  • Each RIME “level” is a way of asking, does the student fulfill that promise

  • Making a diagnosis, not “giving” it.

  • “Objective”

  • Does it affect teacher ratings?


Example construct validity grade distributions univ of utah l.jpg
Example: Construct Validity performance.Grade Distributions Univ. of Utah

Low Pass Pass HP Honors

Battistone Acad. Med. , 2001


Grade distributions univ of utah after rime methods l.jpg
Grade Distributions Univ. of Utah performance.after RIME methods

O R I M E

Battistone Acad. Med. , 2001


Case using rime l.jpg
Case using RIME: performance.

W.O., student, “presents” Mrs. Jones:

  • 45yo woman with acute lower back pain

  • gives detailed description clinical picture suggestive of acute lumbar strain

  •  through physical examination,

    • blood pressure 130/80, heart rate 80

    • left-sided para-spinal tenderness L2 – L5


Case using rime continued l.jpg
Case using RIME (continued) performance.

  • while student is writing up findings in the patient’s record, you interview and examine patient.

  • Mrs. Jones asks: “Doctor, can you take my blood pressure since no one has?”

  • at what “RIME” level is this student?



Complimentary approaches l.jpg

Analytic performance.(domains)

Attitude

Skills

Knowledge

Synthetic (“steps”)

(non-reporter)

Reporter

Interpreter

Manager/ Educator

Complimentary Approaches

ACGME “Professionalism”


The rime rhythm is familiar l.jpg
The RIME rhythm is familiar: performance.

…….S.0….. ….A…………….P………….

H&P………….

Assessment..

Plan………….

Reporter

Interpreter

Manager/ Educator


Slide63 l.jpg

Reliability versus validity performance.

X X X

X X

X X X

X X X

X X X X

Reliable

Valid


Reliability l.jpg
reliability performance.

  • Computed estimate of whether an assessment tool is testing a single construct.

  • A reliability of 0.8 is considered needed for high stakes decisions (80% signal).


Typical reliabilities l.jpg
Typical Reliabilities performance.

exam type alpha

  • “shelf”-100 MCQ 0.75 - .8

  • Step 1 USMLE 0.9 - .95

  • OSCE (6 stations) 0.5 - 0.6

  • OSCE (12 stations) 0.7 - 0.9


Third year performance level grade l.jpg

(Observer) Low Pass performance.

Reporter Pass

Interpreter High Pass

Manager/ Honors Educator

D 1.0

C 2.0

B 3.0

A 4.0

Third-year performance Level = Grade


Usu medicine clerkship l.jpg
USU Medicine clerkship performance.

(students) Reliability

n = 467 0.83

Roop, Amer J Med, 2001


Slide68 l.jpg

validity performance.

What can you infer from your evaluation?

Are you measuring what is important?

X X X

X X X X

Valid


Edp evaluation system l.jpg
EDP Evaluation System performance.

  • RIME Vocabulary

  • Formal Evaluation Sessions

    • Sit down with teachers every few weeks

    • Noel, J Med Ed, 1987 (detecting marginal students)

    • “Frame of reference training”


Slide70 l.jpg
Content Validity: Detecting Deficiencies in Professionalism performance.% of professionalism domains rated less than acceptable

Hemmer, Academic Medicine, 2000.


Predictive validity sensitivity in predicting internship problems pgy1 supervisor surveys l.jpg
Predictive Validity: Sensitivity in predicting Internship Problems(PGY1 supervisor surveys)

USU classes

of 86 - 93

percent

Low Ratings Bad Comment

Lavin, Academic Medicine 1998


Acceptance 2 medicine l.jpg
Acceptance (2): Medicine Problems

  • 93% teachers’ evaluations

  • 81% NBME subject examination

  • 42 % RIME vocabulary

  • 32% OSCE

  • 22% direct observation (mini-CEX)

Hemmer, Teach Learn Med, 2007


Limitations of our system l.jpg
Limitations of our system Problems

  • RIME is sometimes taken as developmental scheme.

  • The analytic model is very strong, and RIME is used for cognitive growth only.

  • Wanting to skip the Evaluation Sessions - they take time!


Basics innovations l.jpg

Clinicians are good diagnosticians Problems

We are mentors for our students

We promise society duty and expertise

Descriptors provide patterns and exemplars

Descriptive frameworks allow behavioral feedback

Synthetic frameworks make this simpler

Basics Innovations


Strategy75 l.jpg
strategy Problems

  • Simplicity leads to acceptance and use

  • Acceptance to consistency

  • Consistency to fairness


Slide76 l.jpg

Thanks for coming! Problems

[email protected]


ad