Assessing knowledge and performance
This presentation is the property of its rightful owner.
Sponsored Links
1 / 42

Assessing Knowledge and Performance PowerPoint PPT Presentation


  • 82 Views
  • Uploaded on
  • Presentation posted in: General

Assessing Knowledge and Performance. John Littlefield University of Texas Health Science Center at San Antonio. Recall a medical student or resident whose performance made you uneasy. What behavior or event made you uneasy? What action did you take?

Download Presentation

Assessing Knowledge and Performance

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Assessing knowledge and performance

Assessing Knowledge and Performance

John Littlefield

University of Texas Health Science Center

at San Antonio


Recall a medical student or resident whose performance made you uneasy

Recall a medical student or resident whose performance made you uneasy.

  • What behavior or event made you uneasy?

  • What action did you take?

    a.Talk with faculty colleagues about your concerns

    b.Write a candid performance appraisal and send it to the course/residency director

    3.If you wrote a candid appraisal, did an administrative actionoccur related to the student/ resident?


Goals assessment of knowledge and performance

Goals: Assessment of Knowledge and Performance

  • Clarify two distinct uses for assessments of clinical knowledge and performance

  • Define two aspects of validity for all assessment methods

  • Compare and contrast 6 techniques for assessing clinical knowledge and performance

  • Identify poorly written multiple choice test items

  • Write a key features test item

  • Describe a role for narrative comments in scoring interactions with Standardized Patients

  • Describe three elements of a clinical performance assessment system

  • Critique a clinical performance assessment system that you use


Agenda assessment of knowledge and performance

Agenda: Assessment of Knowledge and Performance

  • Exercise: Recall a medical student or resident whose performance made you uneasy

  • Presentation: Quality assurance when assessing clinical knowledge and performance

  • Exercise: Take then critique a multiple choice test

  • Presentation: Key features test items

  • Exercise: Write a key features test item

  • Exercise: Critique a videotaped student-SP interaction

  • Presentation: Widening the lens on SP assessment

  • Exercise: Recommend program director actions based on faculty comments about a resident

  • Presentation: Improving clinical performance assessment systems

  • Exercise: Critique your clinical performance assessment system


Assessing knowledge and performance

Uses for Assessment: Formative vs. Summative

Purpose Feedback for Certification/Grading

Learning

Breadth of Narrow Focus on Broad Focus on

Scope Specific Objectives General Goals

Scoring Explicit Feedback Overall Performance

Learner Affective Little Anxiety Moderate to High

Response Anxiety

Target Audience Learner Society


Validity of knowledge and performance assessments

Validity of Knowledge and Performance Assessments

  • Content - Does the assessment method measure a representative cross-section of student competencies?

  • Structural – Degree to which content and scoring represent competence in each subsection of the performance domain

  • External - Do scores from this assessment method correlate highly with scores from other measures of the same student competencies?

  • Consequential - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment?

  • Generalizability

    • Does the student perform at about the same level across 5 to 7 different patients / case problems?

    • Does the student receive a similar rating from different faculty?

  • Substantive – the context surrounding the assessment evokes the domain of cognitive processes used by a physician


Assessing knowledge and performance

Six Aspects of Assessment Validity Viewed as a Cube

Generalizability

Consequential

Content

External

Structural

Substantive


Generalizability of physician performance on multiple patients

Generalizability of Physician Performance on Multiple Patients


Validity of knowledge and performance assessments1

Validity of Knowledge and Performance Assessments

  • Content - Does the assessment method measure a representative cross-section of student competencies?

  • External - Do scores from this assessment method correlate highly with scores from other measures of the same student competencies?

  • Consequential - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment?

  • Generalizability

    • Does the student perform at about the same level across 5 to 7 different patients / case problems?

    • Does the student receive a similar rating from different faculty?

  • Substantive – the context surrounding the assessment evokes the domain of cognitive processes used by a physician


Substantive aspect of validity four levels of performance assessment 1

Substantive Aspect of Validity: Four Levels of Performance Assessment 1

Does

(Global Rating)

Shows How

( OSCE)

Knows How

(Examination – Oral)

Knows

(Examination – Multiple-choice)

  • Miller, GE. Assessment of clinical skills/competence/performance, Academic Medicine, 65(9), supplement, S63-7, 1990


Assessing knowledge and performance

Comparisons among Six Techniques for Assessment of Clinical Knowledge and Performance________________________________________________________________________________________

MultipleModifiedStructured Multimedia Standardized On-the- Choice Essay Oral Exam Simulation Patients Job PA

  • Knowledge +++ ++ + + + +

  • Interviewing/Interpersonal __ __ + + ++ +

  • Data gathering / history + ++ ++ ++ ++ +

  • Physical exam (technique) __ __ __ + +++ +

  • Reasoning / diagnosis + ++ ++ ++ ++ +

  • Lab utilization/management + ++ ++ ++ ++ +

  • Personal qualities __ __ __ __ __ ++___________________________________________________________________________________

+ = adequate ++ = good +++ = excellent __ = not applicable

Newble D, (1992). Assessing clinical comp. at undergrad level. Med. Educ. V. 26, 504-511


Interim summary of session

Interim Summary of Session

  • Session thus far

    • Two uses of knowledge and performance assessments: Formative and Summative

    • Validity of all assessment techniques

    • Comparisons among 6 assessment techniques

  • Coming up

    • Take and critique a 14 item multiple choice exam

    • Presentation on Key Features items


Knowing and realizing

Knowing and Realizing

  • The mere knowledge of a fact is pale; but when you come to realize your fact, it takes on color. It is all the difference between hearing of a man being stabbed in the heart, and seeing it done.

    Mark Twain, A Connecticutt Yankee in King Arthur’s Court


How are multiple choice items selected for an exam

How are Multiple Choice Items Selected for an Exam?


Sample exam blueprint based on clinical problems

Sample Exam Blueprint based on Clinical Problems

Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.


Key features of a clinical problem

Key Features of a Clinical Problem

  • Definition: Critical steps that must be taken to identify and manage a patient’s problem

    • focuses on a step in which examinees are likely to make an error

    • is a difficult aspect in identifying and managing the problem

  • Example: For a pregnant woman experiencing third-trimester bleeding with no abdominal pain, the physician should:

    • generate placenta previa as the leading diagnosis

    • avoid performing a pelvic examination (may cause bleeding)

    • avoid discharging from clinic or emergency room

    • order coagulation tests and cross-match

Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.


Test items based on a clinical problem and its key features

Test Items based on a Clinical Problem and its Key Features

Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.


Scoring the placenta previa clinical problem

Scoring the Placenta Previa Clinical Problem

  • Key Feature 1: To receive one point, must list placenta previa or one of the following synonyms: marginal placenta or low placental insertion

  • Key Features 2-4: Receive 1/3 point for listing each of the following: 1. Avoid performing a pelvic exam, 2. Avoid discharging from clinic, 3. Order coagulation tests and cross match

  • Total Score for Problem: Add scores for items 1 and 2 and divide by 2 (range: 0 - 1)


Steps to develop a clinical problem based exam

Steps to Develop a Clinical Problem Based Exam

  • Define the domain of clinical problems to be sampled by the exam

  • Develop an exam blueprint to guide selection of clinical problems

  • Develop a key-feature problem for each clinical problem selected

    • define clinical situation for the problem (e.g. single typical problem, life-threatening situation etc.)

    • define key features of the problem

    • select a clinical case to represent the problem and write scenario

    • write exam items for case; in general one item for each key feature

    • select suitable format for each item (e.g., write-in or mcq)

    • develop scoring key for each item

    • pilot test items for item analysis data to guide refinement


Interim summary of session1

Interim Summary of Session

  • Session thus far

    • Two uses of knowledge and performance assessments: Formative and Summative

    • Validity of all assessment techniques

    • Comparisons among 6 assessment techniques

    • Take and critique a 14 item multiple choice exam

    • Write a Key Features item

  • Coming up

    • View and critique a videotaped student-patient interaction as part of an OSCE


Schematic diagram of a 9 station osce

Schematic Diagram of a 9 Station OSCE

Start

1

2

3

4

5

9

End

8

7

6


Widening the lens on sp assessment 1

Widening the Lens on SP Assessment1

  • Traditional scoring of SP assessment focuses on numerical data typically from checklists

  • Dimensions of the SP exam

    • basic science knowledge (organize the information)

    • physical exam skills (memory of routines)

    • establishing a human connection

    • role of the student (appear knowledgeable)

    • existential dimension of the human encounter (balance one’s own beliefs with the patient’s)

  • Clinical competence – mixture of knowledge and feeling, information processing and intuition

1. Rose, M. & Wilkerson, L. Widening the Lens on Standardized Patient Assessment: What the Encounter Can Reveal about the Development of Clinical Competence, Acad. Med. 76(8), 2001.


Interim summary of session2

Interim Summary of Session

  • Session thus far

    • Two uses of knowledge and performance assessments: Formative and Summative

    • Validity of all assessment techniques

    • Comparisons among 6 assessment techniques

    • Take and critique a 14 item multiple choice exam

    • Write a Key Features item

    • View and critique a videotaped student-patient interaction

    • Presentation: Widening the lens on SP assessment

  • Coming up

    • Improving clinical performance assessment systems


Dr tough s memo regarding dr will e makit pgy 2

Dr. Tough’s Memo regarding Dr. Will E. Makit (PGY 2)

“The performance of Dr. Makit in General Surgery this month has been completely unsatisfactory. Every member of the clinical faculty who has had any contact with him tells me of his gross incompetence and irresponsibility in clinical situations. This person is an embarrassment to our school and our training program. I spoke to him about his performance after I talked with you several weeks ago and he told me that he would improve it. There was no evidence that he made any effort to improve. There is no way this can be considered a completed or satisfactory rotation in General Surgery. In fact, he is the most unsatisfactory resident who has rotated through our service in the last five years, and his behavior is an appalling example to the rest of our housestaff.” *************************************************************

Your1.Refer the problem to the Resident Education Action?Committee for an administrative decision.

2.Assign Dr. Makit to a rotation with Dr. Insightful as the Attending Faculty.


Dr insightful s phone comments regarding dr makit

Dr. Insightful’s Phone Comments regarding Dr. Makit


Resident performance assessment system

Resident Performance Assessment System

Organizational Infrastructure


A department s organizational infrastructure

A. Department’s Organizational Infrastructure

  • Department head emphasizes completing and returning PA forms

  • Consequences for evaluators who don’t complete PA forms

  • PA form is brief ( < 10 competencies)

  • Don’t request pass/fail judgment by individual faculty

  • Evaluators trained to use PA form & criteria

  • Evaluators believe they will be supported when writing honest appraisals

  • Specific staff assigned to monitor compliance in returning forms

  • Program Director alerted immediately when a returned form reflects cautionary info

Organizational Infrastructure


Evaluator role in a performance assessment system

Evaluator Role in a Performance Assessment System


B evaluator role communicate expectations and observe performance

B. Evaluator Role: Communicate Expectations and Observe Performance

  • Communicate Expectations

    • Consensus among evaluators about service and education expectations

    • Residents are crystal clear about service and education expectations

  • Observe Performance

    • Evaluators observe resident multiple times before completing PA form

    • Appraise only performance directly observed

    • Other staff (e.g., nurses) complete PA forms

Communicate Expectations

Observe Performance


B evaluator s role interpret and judge performance

B.Evaluator’s Role: Interpret and Judge Performance

  • Evaluators agree on what behaviors constitute outstanding, ‘average’, and marginal performance

  • When facing a marginal resident, evaluators record rationale for their judgment and info surrounding the event

  • Evaluators record their interpretation of the performance soon after behavior occurs

diagnose performance (quality wnl ?)


B evaluator s role coach resident

B.Evaluator’s Role: Coach Resident

  • Evaluators aware of difference between corrective feedback, criticism and compliments

  • Faculty actively coach residents in timely manner

  • Residents are encouraged to ask for feedback

  • Evaluators regularly invite self-assessment from residents before giving feedback

Coach Resident


B evaluator s role communicate performance information and complete pa form

B. Evaluator’s Role: Communicate Performance Information and Complete PA Form

  • Communicate Performance Info

    • Communicate incidents that are significantly negative or positive

    • Document in writing even single instances of poor or inappropriate behavior

  • Complete PA Form

    • Evaluators write specific narrative comments on PA forms

    • Evaluators forward completed PA forms to Director in timely way

Communicate performance information (to whom ?)

Complete PA Form


Program director role in a clinical performance assessment system

Program Director Role in a Clinical Performance Assessment System


C program director s role monitor and interpret appraisals

C. Program Director’s Role: Monitor and Interpret Appraisals

  • Recognize evaluator rating patterns (stringent vs. lenient) to accurately interpret PA

  • Contact evaluators to elicit narrative info when absent to substantiate a marginal PA

  • Store PA forms in residents’ files in a timely manner

  • Summarize PA data to facilitate decision making by Resident Education Committee

  • Keep longitudinal records of PA data to develop norms for the PA form


C program director s role committee decision

C. Program Director’s Role: Committee Decision

  • PA decisions are a collaborative process involving multiple faculty

  • Seven or more PA forms per resident are available when admin decisions are made

  • Sufficient written narrative documentation is available when admin decisions are made

committee decision


C program director s role formally inform resident

C. Program Director’s Role:Formally Inform Resident

  • Residents are given a summary of their performance every six months

  • Evaluators have written guidelines outlining what must legally be in a probation letter

  • Evaluators know what documentation is needed to ensure adequate due process

  • Each resident receives an end of program performance evaluation

inform resident


Formative evaluation diagnostic checklist for resident performance assessment system

Formative Evaluation: Diagnostic Checklist for Resident Performance Assessment System


Assessing knowledge and performance

Research: Improving Resident Performance Appraisals 1

  • Organizational Infrastructure

    • Discussed PA problems at department meetings

    • Appointed a task force to review PA problems and propose solutions

    • Revised old appraisal form

    • Pilot-tested and adapted the new appraisal form

  • Evaluator Role

    • Provided evaluators with examples of behaviorally-specific comments

  • Results

    • Increased # of forms returned, # forms with behaviorally-specific comments, and # of administrative actions by prog.

1. Littlefield, J. and Terrell, C. Improving the quality of resident performance appraisals, Academic Medicine, 1997: 72(10) Supplement, S45-47.


Assessing knowledge and performance

Research: Improving Written Comments byFaculty Attendings1

  • Organizational Infrastructure

    • Conducted a 20 minute educational sessions on evaluation and feedback

    • 3 by 5 reminder card and diary

  • Results

    • Increased written comments specific to defined dimensions of competence

    • Residents rated quantity of feedback higher and were more likely to make changes in clinical management of patients

1. Holmboe, E., et.al. Effectiveness of a focused educational intervention on resident evaluations from faculty. J. Gen Intern Med. 2001: 16:427-34.


Research summative evaluation of a pa system 1 2

Research: Summative Evaluation of a PA System1,2

  • 1. Littlefield, J.H., Paukert, J., Schoolfield, J. Quality assurance data for resident global performance ratings, Academic Med., 76(10), supp., S102-04, 2001.

  • 2. Paukert, J. et. al., Improving quality assurance data for resident subjective performance assessment, manuscript in preparation


Recall a medical student or resident whose performance made you uneasy1

Recall a medical student or resident whose performance made you uneasy.

  • What behavior or event made you uneasy?

  • What action did you take?

    a.Talk with faculty colleagues about your concerns

    b.Write a candid performance appraisal and send it to the course/residency director

    3.If you wrote a candid appraisal, did an administrative actionoccur related to the student/ resident?


Goals assessment of knowledge performance

Goals: Assessment of Knowledge & Performance

  • Clarify two distinct uses for assessments of clinical knowledge and performance

  • Define two aspects of validity for all assessment methods

  • Compare and contrast 6 techniques for assessing clinical knowledge and performance

  • Identify poorly written multiple choice test items

  • Write a key features test item

  • Describe a role for narrative comments in scoring interactions with Standardized Patients

  • Describe three elements of a clinical performance assessment system

  • Critique a clinical performance assessment system that you use


  • Login