Using an institutional report card to support evidence based gme decision making
This presentation is the property of its rightful owner.
Sponsored Links
1 / 42

Using an Institutional Report Card to Support Evidence-Based GME Decision Making PowerPoint PPT Presentation


  • 54 Views
  • Uploaded on
  • Presentation posted in: General

Using an Institutional Report Card to Support Evidence-Based GME Decision Making. Conference Session: SES46 20101ACGME Annual Education Conference Ann Dohn, MA, DIO, Alice Edler, MD, MPH, MA ( Educ ), Nancy Piro, PhD, and Bardia Behravesh, EdD , Program Managers/Ed Specialists

Download Presentation

Using an Institutional Report Card to Support Evidence-Based GME Decision Making

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Using an institutional report card to support evidence based gme decision making

Using an Institutional Report Card to Support Evidence-Based GME Decision Making

Conference Session: SES46

20101ACGME Annual Education Conference

Ann Dohn, MA, DIO, Alice Edler, MD, MPH, MA (Educ),

Nancy Piro, PhD, and Bardia Behravesh, EdD,

Program Managers/Ed Specialists

Department of Graduate Medical Education

Stanford Hospital & Clinics


Agenda

AGENDA

  • This workshop session will:

    • Discuss the need for comparative programmatic evaluation

    • Describe existing “report card/scorecard” models from industry that could be used in our institutions

    • Present an example of the “Stanford Report Card” for comparative program evaluation

    • Facilitate exercises that will support participants in developing “report cards” and their uses based on individual needs.


Session objectives

Session Objectives

  • At the end of this session, participants will be able to:

    • Understand the basis of organizational performance assessment

    • Describe some different models for organizational report cards

    • Identify key areas to include in GME “Report Cards”

    • Understand key considerations for using and distributing programmatic evaluation data.


Stanford background

Stanford Background

Stanford University Medical Center currently sponsors 82 ACGME-accredited training programs with over 1000 enrolled residents and fellows.


Stanford university medical center mission

Stanford University Medical Center Mission

  • Dedication to pursuing the highest quality of patient care and graduate medical education, recognizing that one of its major responsibilities is the provision of organized educational programs.

    • Support of quality graduate medical education programs and excellence in residency training and research.

    • Guidance and supervision of the resident while facilitating the resident’s professional and personal development and ensuring safe and appropriate care for patients.

    • Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.

    • Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.


Why do this

Why Do This?

  • We know we’re great…..our residents love us!!

    • Every Program Director will tell you so…


Can we wait

Can We Wait?

  • Can we afford to be slow moving?

  • Can we wait for ACGME site visits?

  • Can we wait for Internal Reviews?


Using an institutional report card to support evidence based gme decision making

But….

  • Our goal is a five year ACGME cycle

    • Internal reviews at the 2-1/2 year mark…

    • A lot can happen in 2-1/2 years


We think we need this

We Think We Need This

  • ACGME and Institutions are increasingly holding DIOs and GME Committees accountable for their utilization of institutional resources.

  • Actions / decisions must be based on documented real-time analyses of needs.


Using an institutional report card to support evidence based gme decision making

DIOs

need to be able to make evidence-based programmatic decisions based on comparative data


Few models exist today for gme

Few Models Exist Today For GME

  • Prior to the era of outcome competency, educational quality was perceived solely as test score measurement and credentialing accomplishment.

    • with the introduction of core competency education medical educators, learners and patients are demanding a more holistic approach to quality medical training.


Few models exist today for gme1

Few Models Exist TodayFor GME

  • The concept of Institutional Accountability is relatively new.

    • Until the ACGME Outcome Project, there was no centralized curriculum oversight in GME, unlike medical schools or UME.


The report card vision

The Report Card Vision

  • In 2005, Stanford hired its first PhD in GME

    • The vision was to develop tools to construct evidence based decision-making for Graduate Medical Education consistent with our mission

    • “We needed a Report Card”…


Using an institutional report card to support evidence based gme decision making

SIGH….

  • It wasn’t as easy as first thought!


Our first attempt

Our First Attempt …


Background on institutional report cards

Background on InstitutionalReport Cards

  • Government and Industry Models

    • Multiple models exist and can be used as per specific purpose:

      • GRPA (Government Performance and Results Act)

      • Organizational Report Cards

      • Balanced Scorecard

      • Benchmarking

      • Program Evaluations

      • Social Indicators


Report card vs balanced scorecard

Report Card vs. Balanced Scorecard


Which model to choose

Which Model to Choose?

  • We needed a model that was:

    • Organizationally focused and managed

    • Track record of effective use

    • Fit our existing structure with multiple programs and organizations

    • Flexible enough to be adapted for use on an annual basis – not an accreditation cycle-Regular Data Collection

    • “Easily digestible” internal and external measurement dimensions


Our choice

Our Choice

Balanced Scorecard Framework

in an Organization

Report Card

(Scorecard) Tool

Best of Both Worlds


Stanford hospital clinics report card

Stanford Hospital & Clinics Report Card

  • The SHC Report Card is built on the Balanced Scorecard conceptual framework for translating an organization’s vision into a set of performance indicators distributed among four perspectives adapted for GME:

    • Resident Perception Measurements

    • Program Processes

    • Learning Outcomes

    • Financial/Growth


Stanford hospital clinics report card1

Stanford Hospital & Clinics Report Card

  • Resident Perception Measurements

    “Guidance and supervision of the resident while facilitating the resident’s professional and personal development and ensuring safe and appropriate care for patients.”

    2. Program Processes

    “Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.”


Stanford hospital clinics report card2

Stanford Hospital & Clinics Report Card

3. Learning Outcomes

“Support of quality graduate medical education programs and excellence in residency training and research.”

4. Financial/Growth

“Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.”


The balanced scorecard approach

The Balanced Scorecard Approach

  • The Balanced Scorecard is a performance measurement and performance management system developed by Robert Kaplan and David Norton (1992, 1996)

    • adopted by a wide range of leading edge organizations, both public and private.

      (“The Balanced Scorecard--Measures That Drive Performance,” Harvard Business Review, Jan-Feb 1992; and “The Balanced Scorecard-Translating Strategy into Action,” Harvard Business School Press, 1996)


Stanford hospital clinics report card3

Stanford Hospital & Clinics Report Card

  • Indicators are designed to measure SHC’s progress toward achieving its vision; other indicators are designed to measure the long term drivers of success.

  • Through the balanced scorecard, SHC :

    • Monitors its current performance (finances, resident satisfaction, learning outcomes and program process results)

    • Monitors its efforts to improve processes, educate residents

    • Enhances its ability to grow, learn and improve the quality of its fellowship and residency educational programs.


Balanced scorecard strategic perspectives

Balanced Scorecard Strategic Perspectives

Resident

How do our residents see us?

Are our programs excelling?

Institutional / Financial Growth

Mission

Vision

Strategy

Program

Processes

Are we putting our resources in the right places?

Do we continue to improve (outcomes)?

Learning


Measurement across the continuum

Measurement Across The Continuum

  • PRE: Measuring events that occur before the trainee arrives

    • NRMP Results

  • PERI: During Residency

    • ACGME Survey

  • POST: After they have left training to start their career.

    • Alumni Survey


Selection of report card measures

Selection of Report Card Measures

  • # Applicants/Open Positions

  • Match Depth

  • % Top Medical Schools

  • GME Internal HS Survey

  • Overall Satisfaction

  • Recommendation of Program

  • Teaching Quality

  • Curriculum Quality

  • Educational Leadership

  • Wellness Index

  • ACGME Survey

  • Compliant Responses

  • Alumni Survey

  • Faculty Eval of program

  • Resident Eval of program

  • Faculty Publications

  • Duty Hr Violations

  • ACGME Cycle Length

  • # ACGME Citations

  • Core Competency Self-Assessment

  • ITSE Scores

  • Annual Resident Publications

  • Annual Resident Presentations

  • # Safety Incident Reports

  • Specialty Board Scores

  • Core Competency Post Assessment

  • # Res in Program

  • Grants Awarded

  • Subspecialties/Program

  • Physical Space/Facilities


Voice of the residents

“Voice of the Residents”


Program processes

Program Processes


Program matrix

Program Matrix


Learning outcomes

Learning Outcomes


Financial growth

Financial / Growth


Case study stanford

Case Study - Stanford

  • How the DIO uses the Report Card


How do we use this data

How Do We Use this Data?

  • Look at Indicators that are Resident Driven – “Voice of the Resident”

    • Is there a discrepancy between the voice of the resident and the other indicators?

      • Would the majority of the residents not choose the program again yet the Board Scores are high?


How do we use this data1

How Do We Use this Data?

  • How Do the Programs Compare Against Each Other?

  • How do they compare against their ACGME Cycles?


What s next

What’s Next?

  • GME Staff Brainstorming Session

  • The Why’s – Why are programs where they are?

  • Where do we need to focus our resources?


Presenting the data

Presenting the Data

  • Program Directors Monthly Forum

    • Protect the Name of the Program

      • Growth and Change not Blame


Presenting the data1

Presenting the Data

  • Individual Meetings with Program Directors

    • Share Complete Data


Action planning

Action Planning

  • GME Staff working with Program Directors

  • Sharing Findings with GMEC and Administration


Political fallout

Political Fallout

  • No Program Director wants to be at the bottom…

  • Defensiveness

  • Bragging Rights


And by the way this will help you answer

And by the way… this will help you answer:

COMMON INSTITUTIONAL REVIEW DOCUMENT

Question 30b:

“Describe how the sponsoring institution monitors that each program provides effective educational experiences for residents that lead to measureable achievement of educational outcomes of the ACGME competencies.”


Questions

Questions


  • Login