Observable Practice Activities (OPAs)
This presentation is the property of its rightful owner.
Sponsored Links
1 / 51

Observable Practice Activities (OPAs) PowerPoint PPT Presentation


  • 90 Views
  • Uploaded on
  • Presentation posted in: General

Observable Practice Activities (OPAs). Content-Based (COPA). Process-Based (POPA). Mid/End of Rotation Feedback (formative) Entrustment (1-5). Mapping. Mapping. Curricular Milestones. End-of-training EPAs. Reporting Milestones. External Reporting NAS report .

Download Presentation

Observable Practice Activities (OPAs)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Observable practice activities opas

Observable Practice Activities (OPAs)

Content-Based (COPA)

Process-Based (POPA)

Mid/End of Rotation Feedback (formative)

Entrustment (1-5)

Mapping

Mapping

Curricular

Milestones

End-of-training

EPAs

Reporting

Milestones

External Reporting

NAS report

Longitudinal Assessment

(summative)

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Mapping

Mapping

  • We mapped with the following in mind:

    • when assessing a trainee on a given OPA, would the evaluator feel as if they were also assessing the mapped milestones?

    • We removed all soft calls

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Opa minimize unfamiliar terms during patient encounters

OPA: Minimize unfamiliar terms during patient encounters

Milestones

ICS-A2 - Effectively use verbal and nonverbal skills to create rapport with patients/families

ICS-D1 - Deliver appropriate, succinct, hypothesis-driven oral presentations

SBP-B2 - Work effectively as a member within the inter-professional team to ensure safe patient care


Observable practice activities opas

GIM

PGY-1

Wards


Observable practice activities opas

COPA: Identify causes of delirium

Reporting Milestones

PC-1 - Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s).

MK-1   Clinical knowledge


Copa manage exacerbations of obstructive lung disease

COPA: Manage exacerbations of obstructive lung disease

Reporting Milestones

PC-2   Develops and achieves comprehensive management plan for each patient.

PC-3   Manages patients with progressive responsibility and independence.

MK-1   Clinical knowledge

MK-2   Knowledge of diagnostic testing and procedures


Copa initiate cost effective workup for anemia

COPA: Initiate cost-effective workup for anemia

Reporting Milestones

PC-1   Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s)

MK-1   Clinical knowledge

MK-2   Knowledge of diagnostic testing and procedures

SBP-3 Identifies forces that impact the cost of health care, and advocates for, and practices cost-effective care


C opa assess and treat pain

COPA: Assess and treat pain

Reporting Milestones

PC-2 Develops and achieves comprehensive management plan for each patient

PC-3 Manages patients with progressive responsibility and independence

MK-1   Clinical knowledge

PROF-3 Responds to each patient’s unique characteristics and needs


Popa demonstrate shared decision making with the patient

POPA: Demonstrate shared decision-making with the patient

Reporting Milestones

PC-2 Develops and achieves comprehensive management plan for each patient

PROF-1 - Has professional and respectful interactions with patients, caregivers and members of the interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and support personnel)

PROF-3 Responds to each patient’s unique characteristics and needs

ICS-1 - Communicates effectively with patients and caregivers.


Popa use feedback to improve performance

POPA: Use feedback to improve performance

Reporting Milestones

PBLI-3 - Learns and improves via feedback

PROF-4 - Exhibits integrity and ethical behavior in professional conduct.


Mapping opas to milestones

Mapping OPAs to Milestones

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Observable practice activities opas

Mapping OPAs to Milestones

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Observable practice activities opas

Mapping OPAs to Milestones

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Observable practice activities opas

Mapping OPAs to Milestones

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Observable practice activities opas

Mapping OPAs to Milestones

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Observable practice activities opas

Patient Care

Patient Care

Traditional Competency-Based Evaluations

Manage Vent

Read EKG

Diagnose Hepatitis

Titrate Insulin

OPA Based Evaluations


Observable practice activities opas

Titrate Insulin

Read EKG

Manage Vent

Diagnose Hepatitis

The Milestones are the apples to be compared


Observable practice activities opas

Approximately 3000 Assessments


Observable practice activities opas

Reporting Milestones

Resident A

Resident B

Resident A

Resident B

Resident C

Resident C


Observable practice activities opas

Reporting Milestones

Resident E

Resident D


Observable practice activities opas

Reporting Milestones

Resident F

Resident F


Observable practice activities opas

6 nurses


Observable practice activities opas

Reporting Milestones (ACGME)


Observable practice activities opas

Example: PGY-3 30 Month NAS Report

1 1.5 2 2.5 3 3.5 4 4.5 5


Observable practice activities opas

NAS report Jul-Dec 2013


Observable practice activities opas

Single PGY-1 - OPAs from Jul 1st to Feb 28th

Entire Residency – All OPAs from Jul 1st to Feb 28th


Observable practice activities opas

Trend line of all PGY-1 Milestone Assessments By Month n = 105721

Standard Deviation of all PGY-1 Milestones Assessments by Month


Validity

Validity?

  • Validity – how well one can trust the results of a test as interpreted for a specific purpose1

    • Not a property of the instrument, but of interpretation

1Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422


Validity1

Validity?

  • Content: do instrument items represent the construct?

  • Response Process: are the construct and the thought processes of observers the same?

  • Reliability – internal consistency, temporal stability, inter-rater reliability, generalizability

  • Consequences: do scores really make a difference?

  • Relations to other variables

1Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422


Observable practice activities opas

R3 Class Milestone Average Jul 1st to Feb 1st

Selected to be Chief Resident

Had issues brought before competency committee

Patient Care

Communication

Professionalism


Observable practice activities opas

2012

2013


Research directions

Research Directions

  • Study the progression of assessment entrustment decisions made over time for internal medicine residents by faculty physicians, peer physicians, and allied health providers

  • Identify assessment biases present in these entrustment decisions, including bias associated with age, sex, resident year of training, attending experience level, or job description of assessor

  • Determine validity of assessment entrustment decisions made over time for internal medicine residents in relation to other variables such as standardized test scores, and other clinical assessments


Observable practice activities opas

Reporting

Milestones

Narratives

OPAs

Level Of Entrustment

1 2 3 4 5

Rides in Driveway

Rides Bicycle Safely

Falls off Bike

Rides 1000 feet without Stopping

Rides on Sidewalk

Rides on Quiet Street

Level Of Entrustment

1 2 3 4 5

Rides on Quiet Road

Rides in Traffic

Rides in Rush Hour Traffic

Rides to work on time

Rides Tour de France

Rides in Triathlon


Observable practice activities opas

Reporting

Milestones/EPAs

Narratives

OPAs

Level Of Entrustment

1 2 3 4 5

Develops and achieves comprehensive management plan for each patient. (PC2)

Care plans are consistently inappropriate or inaccurate

Titrate cardiac medications

Manage pancreatitis

Inconsistently develops an appropriate care plan

Titrate insulin based on glucose readings

Adjust short and long acting narcotics for cancer and sickle cell patients

Consistently develops appropriate care plan

Level Of Entrustment

1 2 3 4 5

Appropriately modifies care plans based on patient’s clinical course, additional data, and patient preferences

Manage parathyroid abnormalities in the renal patient

Deliver appropriate goal-directed therapy for severe sepsis

Role models and teaches complex and patient-centered care


Observable practice activities opas

Observable Practice Activities (OPAs)

Content-Based (COPA)

Process-Based (POPA)

Mid/End of Rotation Feedback (formative)

Entrustment (1-5)

Mapping

Mapping

Curricular

Milestones

End-of-training

EPAs

Reporting

Milestones

External Reporting

NAS report

Longitudinal Assessment

(summative)

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518


Team effort

TEAM EFFORT

ResidencyProgram Staff

Dell

Diers

Warm

Mathis

Fichtenbaum

Mueller

O’Toole

Pai

Schauer

Tolentino

Lee

Ashbrook

Held

Lee

Wood

Wexler

ResidencyEducationCoordinators

Kamath

Schoch

Lenz

Khan

Robertson

Houk

Falciglia

Chaudhary

Cohen


Discussion questions from the survey

Discussion questions from the survey

  • Is it possible to have a generic map that can be used for both Medicine and Peds?

    • You could design an excel solution which Eric initially did before medhub figured out how to do it for us

    • Other ideas?


Discussion questions from the survey1

Discussion questions from the survey

  • Specific directions for Med-Peds reporting

    • Will be straightforward with the same forms

    • ACGME sent me a copy of the reporting form when I called them and asked them to; it is literally just the 2 sets put together into 1 form


What about similar milestones

What about similar milestones?

  • Some of this is subject to opinion, but about 8-10 are largely the same

  • What do I mean?

  • This is split into PC, MK and SBP as one slide and PBLI, Prof, and ICS as second slide

  • ALERT: COMPLICATED VISUALS!!!


Pc mk sbp

PC, MK, SBP


Bottom line

Bottom line

  • 22 IM milestones

  • 21 Peds milestones

  • 43 total

  • Of these, 9 have significant overlap, so there are actually 43 minus 9 or 34 SEPARATE milestones to report


Discussion questions from the survey2

Discussion questions from the survey

  • What about overlapping milestones?

    • Since the goal is to establish whether residents are progressing and competent, is there a reason not to use the benefit of evaluations from both sides together?


Questions

Questions?


  • Login