1 / 18

Faculty Development Session on Evaluations

Faculty Development Session on Evaluations. Graduate Medical Education Ann Dohn, MA, Designated Institutional Official (DIO) Nancy Piro, PhD Program Manger/Education Specialist Kim Walker, PhD Program Manger/Education Specialist. ACGME Requirements: Seven Domains. I. Institution

richelle
Download Presentation

Faculty Development Session on Evaluations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Faculty Development Session on Evaluations Graduate Medical Education Ann Dohn, MA, Designated Institutional Official (DIO) Nancy Piro, PhD Program Manger/Education Specialist Kim Walker, PhD Program Manger/Education Specialist

  2. ACGME Requirements: Seven Domains I. Institution II. Program Personnel & Resources III. Resident Appointments IV. Educational Program • Evaluation • Resident • Faculty • Program VI. Resident Duty Hours VII. Experimentation & Innovation

  3. Faculty Development SessionEVALUATIONS Our focus today: How do we eliminate unintended bias from our evaluation process?

  4. What is cognitive bias… • Cognitive bias is distortion in the way we perceive reality / information. • Response bias is a type of cognitive bias which can affect the results of an evaluation if evaluators answer questions in the way they think they are designed to be answered, or with a positive or negative bias toward the fellow being evaluated

  5. Where does response bias occur? • Response bias most occurs most often in the wording of the question. • Response bias is present when a question contains a leading phrase or words. • Response bias can also occur in rating scales. • Response bias can be in the raters themselves • Central Tendency • Halo Effect • Similarity Effect

  6. Examples of Question Bias • Example 1: "I can always talk to my Program Director about residency related problems." • Problem: Terms such as "always" and "never" will bias the response in the opposite direction. • Result: Data will be skewed.

  7. Examples of Question Bias • Example 2: “Career planning resources are available to me and my program director supports my professional aspirations." • Problem: Double-barreled ---resources and aspirations… Respondents may agree with one and not the other. Evaluator cannot make assumptions about which part of the question respondents were rating. • Result: Data is useless.

  8. Examples of Question Bias • Example 3: "Communication in my program is good." • Problem: Question is too broad. If score is less than 100% positive, researcher/evaluator still does not know what aspect of communication needs improvement. • Result: Data is of little or no usefulness.

  9. Rating Scale Bias Competence and knowledge in general medicine. Poor Fair Good Very Good Excellent The data will be artificially skewed in the positive direction with this scale because there are far more (4:1) positive than negative rating options.

  10. Rater/Evaluator Bias Response bias can be in the evaluators themselves • Central Tendency • Similarity Effect • Halo Effect

  11. Beware the Halo Effect • The halo effect refers to a cognitive bias whereby the perception of a particular behavior or trait is influenced by the perception of the former traits in a sequence of interpretations. • Thorndike (1920) was the first to support the halo effect with empirical research. • People seem not to think of other individuals in mixed terms; instead we seem to see each person as roughly good or roughly bad across all categories of measurement.

  12. The Halo Effect and Expectations • The halo effect is involved in Kelley's implicit personality theory • the first traits we recognize in other people influence our interpretation and perception of later ones because of our expectations.

  13. Halo Effect Extends to Products / Marketing The iPod has had positive effects on perceptions of Apple’s other products…

  14. Could this impact our evaluations here? Empirical evidence from our HouseStaff…. A question from the most recent GME HouseStaff survey: “The general feeling in my program is that your ability will be labeled based on your initial performance.” Overall Peds Fellows Overall SHC-LPCH

  15. Reverse Halo Effect • A corollary to the halo effect is the reverse halo effect (devil effect) • individuals, brands or other things judged to have a single undesirable trait are subsequently judged to have many poor traits, allowing a single weak point or negative trait to influence others' perception of the person, brand or other thing in general.

  16. Blind Spots • In the 1970s, the social psychologist Richard Nisbett demonstrated that we may have no awareness of when the halo effect influences us (Nisbett, R.E. and Wilson, T.D., 1977) The problem with Blind Spots is that we are blind to them…

  17. Medical Knowledge Systems-based Practice Patient Care Professionalism Interpersonal & Communication Skills Practice-based Learning & Improvement Know the ACGME Core Competenciesand How You are Evaluating Them

  18. Questions?

More Related