Assessment of knowledge and performance
Download
1 / 29

Assessment of Knowledge and Performance - PowerPoint PPT Presentation


  • 165 Views
  • Updated On :

Assessment of Knowledge and Performance. John Littlefield, PhD University of Texas Health Science Center at San Antonio . Goals: Assessment of Knowledge and Performance. 1. Clarify 2 distinct uses for assessments of clinical knowledge and performance

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Assessment of Knowledge and Performance' - casey


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Assessment of knowledge and performance l.jpg

Assessment of Knowledge and Performance

John Littlefield, PhD

University of Texas

Health Science Center at San Antonio


Goals assessment of knowledge and performance l.jpg
Goals: Assessment of Knowledge and Performance

1. Clarify 2 distinct uses for assessments of clinical knowledge and performance

2. Define 2 aspects of validity for all assessment methods

3. Compare and contrast 3 techniques for assessing clinical knowledge and performance

4. Identify poorly written multiple choice test items and write a key features test item

5. Describe 3 options for scoring OSCE performance

6. Describe three elements of a clinical performance assessment system

7. Critique a clinical performance assessment system that you use

8. Recognize legal issues related to evaluating and dismissing students and residents


Agenda assessment of knowledge and performance l.jpg
Agenda: Assessment of Knowledge and Performance

  • Exercise: Warm-up for assessing knowledge and performance

  • Presentation: Quality assurance when assessing clinical knowledge and performance

  • Exercise: Take then critique a multiple choice test

  • Presentation: Key features test items

  • Exercise: Write a key features test item

  • Presentation: Widening the lens on SP assessment

  • Exercise: Strengths & weaknesses of a clinical performance assessment system that you use

  • Presentation: Improving clinical performance assessment systems

  • Exercise: Critique your clinical performance assessment system

  • Discussion: Legal context for evaluations and dismissals


Warm up exercise l.jpg
Warm-up Exercise

  • What answer did you circle to questions 1.a (4 cards) and 1.b (4 people)?

    2. What concerns do you have about assessing knowledge and performance that you would like addressed?


Slide5 l.jpg

Uses for Assessment: Formative vs. Summative

Purpose Feedback for Certification/Grading

Learning

Breadth of Narrow Focus on Broad Focus on

Scope Specific Objectives General Goals

Scoring Explicit Feedback Overall Performance

Learner Affective Little Anxiety Moderate to High

Response Anxiety

Target Audience Learner Society


Validity of knowledge and performance assessments l.jpg
Validity of Knowledge and Performance Assessments

  • Content - Does the assessment method measure a representative cross-section of student/resident competencies?

  • Internal structure – Do content and scoring focus on a specific clinical competency (e.g., patient care)?

  • Relation to other assessments - Do scores from this assessment correlate highly with other measures of same student competency?

  • Consequences - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment?

  • Generalizability

    • Does the student perform at about the same level across 5 to 7 different patients / case problems?

    • Does the student receive a similar rating from different faculty?

  • Cognitive process – the context surrounding the assessment evokes the domain of cognitive processing used by a physician

1. Standards for Educ. & Psych. Testing, AERA, APA, NCME, 1999, p 11-16.


Slide7 l.jpg

Six Aspects of Assessment Validity Viewed as aCube

Generalizability

Relation to other assessments

Internal structure

Consequences

Content

Cognitive process



Validity of knowledge and performance assessments9 l.jpg
Validity of Knowledge and Performance Assessments as a

  • Content - Does the assessment method measure a representative cross-section of student/resident competencies?

  • Internal structure – Do content and scoring focus on a specific clinical competency (e.g., patient care)?

  • Relation to other assessments - Do scores from this assessment correlate highly with other measures of same student competency?

  • Consequences - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment?

  • Generalizability

    • Does the student perform at about the same level across 5 to 7 different patients / case problems?

    • Does the student receive a similar rating from different faculty?

  • Cognitive process – the context surrounding the assessment evokes the domain of cognitive processing used by a physician

1. Standards for Educ. & Psych. Testing, AERA, APA, NCME, 1999, p 11-16.


Cognitive process aspect of validity four levels of performance assessment 1 l.jpg
Cognitive Process Aspect of Validity: as aFour Levels of Performance Assessment1

Does

(Global Rating)

Shows How

( OSCE)

Knows How

(Examination – Oral)

Knows

(Examination – Multiple-choice)

  • Miller, GE. Assessment of clinical skills/competence/performance, Academic Medicine, 65(9), supplement, 1990,S63-7


Compare and contrast three assessment techniques multiple choice exam osce global ratings l.jpg
Compare and Contrast Three Assessment Techniques as a(multiple choice exam, OSCE, global ratings)

M.C.E. OSCE Global rtgs.

  • Content +++ ++ +

  • Internal structure +++ ++ +

  • Rel. to other assessments + + +

  • Consequences + ++ +

  • Generalizability

    • 5 to 7 case problems +++ ++ +

    • agreement among raters +++ ++ +

  • Cognitive process+ ++ +++

    += adequate ++ = good +++ = excellent


Interim summary of session l.jpg
Interim Summary of Session as a

  • Session thus far

    • Two uses of knowledge and performance assessments: Formative and Summative

    • Validity of all assessment techniques

    • Compare and contrast 3 assessment techniques

  • Coming up

    • Take and critique a 14 item multiple choice exam

    • Presentation on Key Features items



Sample exam blueprint based on clinical problems l.jpg
Sample Exam Blueprint based on Clinical Problems as a

Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.


Key features of a clinical problem 1 l.jpg
Key Features of a Clinical Problem as a1

  • Definition: Critical steps that must be taken to identify and manage a patient’s problem

    • focuses on a step in which examinees are likely to make an error

    • is a difficult aspect in identifying and managing the problem

  • Example: For a pregnant woman experiencing third-trimester bleeding with no abdominal pain, the physician should:

    • generate placenta previa as the leading diagnosis

    • avoid performing a pelvic examination (may cause bleeding)

    • avoid discharging from clinic or emergency room

    • order coagulation tests and cross-match

1. Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.



Scoring the placenta previa clinical problem l.jpg
Scoring the Placenta Previa Clinical Problem as a

  • Key Feature 1: To receive one point, must list placenta previa or one of the following synonyms: marginal placenta or low placental insertion

  • Key Features 2-4: Receive 1/3 point for listing each of the following: 1. Avoid performing a pelvic exam, 2. Avoid discharging from clinic, 3. Order coagulation tests and cross match

  • Total Score for Problem: Add scores for items 1 and 2 and divide by 2 (range: 0 - 1)


Steps to develop a clinical problem based exam l.jpg
Steps to Develop a Clinical-Problem Based Exam as a

  • Define the domain of clinical problems to be sampled by the exam

  • Develop an exam blueprint to guide selection of clinical problems

  • Develop a key-feature problem for each clinical problem selected

    • define clinical situation for the problem (e.g. single typical problem, life-threatening situation etc.)

    • define key features of the problem

    • select a clinical case to represent the problem and write scenario

    • write exam items for case; in general one item for each key feature

    • select suitable format for each item (e.g., write-in or mcq)

    • develop scoring key for each item

    • pilot test items for item analysis data to guide refinement


Interim summary of session19 l.jpg
Interim Summary of Session as a

  • Session thus far

    • Two uses of knowledge and performance assessments: Formative and Summative

    • Validity of all assessment techniques

    • Compare and contrast three assessment techniques

    • Take and critique a 14 item multiple choice exam

    • Write a Key Features item

  • Coming up

    • Scoring performance on an SP exam


Schematic diagram of a 9 station osce l.jpg
Schematic Diagram of a 9 Station OSCE as a

Start

1

2

3

4

5

9

End

8

7

6


Scoring osce performance l.jpg
Scoring OSCE Performance as a

  • Traditional scoring of SP assessment focuses on numerical data typically from checklists

  • Checklist scoring may not accurately assess clinical performance quality of residents and expert clinicians 1

  • Dimensions of the SP exam 2

    • basic science knowledge (organize the information)

    • physical exam skills (memory of routines)

    • establishing a human connection

    • role of the student (appear knowledgeable)

    • existential dimension of the human encounter (balance one’s own beliefs with the patient’s)

  • Clinical competence – mixture of knowledge and feeling, information processing and intuition

  • Hodges et. al. OSCE checklists do not capture increasing levels of expertise, Acad. Med. 74(10), 1999, 1129-34. 2. Rose & Wilkerson. Widening the lens on SP assessment: What the encounter can reveal about development of clinical competence, Acad. Med. 76(8), 2001, 856-59.


Interim summary of session22 l.jpg
Interim Summary of Session as a

  • Session thus far

    • Two uses of knowledge and performance assessments: Formative and Summative

    • Validity of all assessment techniques

    • Compare and contrast three assessment techniques

    • Take and critique a 14 item multiple choice exam

    • Write a Key Features test item

    • Use global ratings and narrative comments when scoring OSCE performance

  • Coming up

    • Improving clinical performance assessment systems


Recall a student resident whose clinical performance made you uneasy l.jpg
Recall a student/resident whose clinical performance made you uneasy

  • What action did you take?

    a. Talk with faculty colleagues about

    your concerns: Yes No

    b. Write a candid performance

    assessment and send it to the course director: Yes No

  • If you wrote a candid assessment, did

    an administrative action occur related

    to the student/resident? Yes No

  • Do you think the performance assess-

    ments in your department’s files reflect

    the facultys’ candid appraisal of

    clinical performance quality? Yes No

    Why do you think this is true?



Diagnostic checklist for clinical performance assessment system l.jpg
Diagnostic Checklist for you uneasyClinical Performance Assessment System


Three year study to improve the quality of resident performance assessment data l.jpg
Three Year Study to Improve the Quality of you uneasyResident Performance Assessment Data

  • What median percentage of each resident’s rotations returned one or more completed forms?

    2. How precise were the scores marked on the returned forms?

    3. What median percentage of each resident’s rotations returned one or more forms with behaviorally-specific written comments?

Littlefield, DaRosa, Paukert et. al. Improving Resident Performance Assessment Data: Numeric Precision & Narrative Specificity, Acad. Med. (in press)



Legal issues related to evaluation and dismissal 1 l.jpg
Legal Issues Related to Evaluation and Dismissal you uneasy1

  • Fair and equitable treatment

  • Due process for academic dismissals

  • Documentation, libel, and defamation

1. Irby & Milam. The legal context for evaluating and dismissing medical students and residents. Acad Med, 64(11),1989, 639-43


Goals assessment of knowledge performance l.jpg
Goals: Assessment of Knowledge & Performance you uneasy

1. Clarify 2 distinct uses for assessments of clinical knowledge and performance

2. Define 2 aspects of validity for all assessment methods

3. Compare and contrast 3 techniques for assessing clinical knowledge and performance

4. Identify poorly written multiple choice test items and write a key features test item

5. Describe 3 options for scoring OSCE performance

6. Describe three elements of a clinical performance assessment system

7. Critique a clinical performance assessment system that you use

8. Recognize legal issues related to evaluating and dismissing students and residents


ad