Using an institutional report card to support evidence based gme decision making
Download
1 / 42

Using an Institutional Report Card to Support Evidence-Based GME Decision Making - PowerPoint PPT Presentation


  • 60 Views
  • Uploaded on

Using an Institutional Report Card to Support Evidence-Based GME Decision Making. Conference Session: SES46 20101ACGME Annual Education Conference Ann Dohn, MA, DIO, Alice Edler, MD, MPH, MA ( Educ ), Nancy Piro, PhD, and Bardia Behravesh, EdD , Program Managers/Ed Specialists

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Using an Institutional Report Card to Support Evidence-Based GME Decision Making' - matthew-mendoza


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Using an institutional report card to support evidence based gme decision making

Using an Institutional Report Card to Support Evidence-Based GME Decision Making

Conference Session: SES46

20101ACGME Annual Education Conference

Ann Dohn, MA, DIO, Alice Edler, MD, MPH, MA (Educ),

Nancy Piro, PhD, and Bardia Behravesh, EdD,

Program Managers/Ed Specialists

Department of Graduate Medical Education

Stanford Hospital & Clinics


Agenda
AGENDA GME Decision Making

  • This workshop session will:

    • Discuss the need for comparative programmatic evaluation

    • Describe existing “report card/scorecard” models from industry that could be used in our institutions

    • Present an example of the “Stanford Report Card” for comparative program evaluation

    • Facilitate exercises that will support participants in developing “report cards” and their uses based on individual needs.


Session objectives
Session Objectives GME Decision Making

  • At the end of this session, participants will be able to:

    • Understand the basis of organizational performance assessment

    • Describe some different models for organizational report cards

    • Identify key areas to include in GME “Report Cards”

    • Understand key considerations for using and distributing programmatic evaluation data.


Stanford background
Stanford Background GME Decision Making

Stanford University Medical Center currently sponsors 82 ACGME-accredited training programs with over 1000 enrolled residents and fellows.


Stanford university medical center mission
Stanford University Medical GME Decision MakingCenter Mission

  • Dedication to pursuing the highest quality of patient care and graduate medical education, recognizing that one of its major responsibilities is the provision of organized educational programs.

    • Support of quality graduate medical education programs and excellence in residency training and research.

    • Guidance and supervision of the resident while facilitating the resident’s professional and personal development and ensuring safe and appropriate care for patients.

    • Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.

    • Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.


Why do this
Why Do This? GME Decision Making

  • We know we’re great…..our residents love us!!

    • Every Program Director will tell you so…


Can we wait
Can We Wait? GME Decision Making

  • Can we afford to be slow moving?

  • Can we wait for ACGME site visits?

  • Can we wait for Internal Reviews?


But…. GME Decision Making

  • Our goal is a five year ACGME cycle

    • Internal reviews at the 2-1/2 year mark…

    • A lot can happen in 2-1/2 years


We think we need this
We Think We Need This GME Decision Making

  • ACGME and Institutions are increasingly holding DIOs and GME Committees accountable for their utilization of institutional resources.

  • Actions / decisions must be based on documented real-time analyses of needs.


DIOs GME Decision Making

need to be able to make evidence-based programmatic decisions based on comparative data


Few models exist today for gme
Few Models Exist GME Decision MakingToday For GME

  • Prior to the era of outcome competency, educational quality was perceived solely as test score measurement and credentialing accomplishment.

    • with the introduction of core competency education medical educators, learners and patients are demanding a more holistic approach to quality medical training.


Few models exist today for gme1
Few Models Exist Today GME Decision MakingFor GME

  • The concept of Institutional Accountability is relatively new.

    • Until the ACGME Outcome Project, there was no centralized curriculum oversight in GME, unlike medical schools or UME.


The report card vision
The Report Card Vision GME Decision Making

  • In 2005, Stanford hired its first PhD in GME

    • The vision was to develop tools to construct evidence based decision-making for Graduate Medical Education consistent with our mission

    • “We needed a Report Card”…


SIGH…. GME Decision Making

  • It wasn’t as easy as first thought!


Our first attempt
Our First Attempt … GME Decision Making


Background on institutional report cards
Background on Institutional GME Decision MakingReport Cards

  • Government and Industry Models

    • Multiple models exist and can be used as per specific purpose:

      • GRPA (Government Performance and Results Act)

      • Organizational Report Cards

      • Balanced Scorecard

      • Benchmarking

      • Program Evaluations

      • Social Indicators



Which model to choose
Which Model to Choose? GME Decision Making

  • We needed a model that was:

    • Organizationally focused and managed

    • Track record of effective use

    • Fit our existing structure with multiple programs and organizations

    • Flexible enough to be adapted for use on an annual basis – not an accreditation cycle-Regular Data Collection

    • “Easily digestible” internal and external measurement dimensions


Our choice
Our Choice GME Decision Making

Balanced Scorecard Framework

in an Organization

Report Card

(Scorecard) Tool

Best of Both Worlds


Stanford hospital clinics report card
Stanford Hospital & Clinics GME Decision MakingReport Card

  • The SHC Report Card is built on the Balanced Scorecard conceptual framework for translating an organization’s vision into a set of performance indicators distributed among four perspectives adapted for GME:

    • Resident Perception Measurements

    • Program Processes

    • Learning Outcomes

    • Financial/Growth


Stanford hospital clinics report card1
Stanford Hospital & Clinics GME Decision MakingReport Card

  • Resident Perception Measurements

    “Guidance and supervision of the resident while facilitating the resident’s professional and personal development and ensuring safe and appropriate care for patients.”

    2. Program Processes

    “Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.”


Stanford hospital clinics report card2
Stanford Hospital & Clinics GME Decision MakingReport Card

3. Learning Outcomes

“Support of quality graduate medical education programs and excellence in residency training and research.”

4. Financial/Growth

“Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.”


The balanced scorecard approach
The Balanced Scorecard Approach GME Decision Making

  • The Balanced Scorecard is a performance measurement and performance management system developed by Robert Kaplan and David Norton (1992, 1996)

    • adopted by a wide range of leading edge organizations, both public and private.

      (“The Balanced Scorecard--Measures That Drive Performance,” Harvard Business Review, Jan-Feb 1992; and “The Balanced Scorecard-Translating Strategy into Action,” Harvard Business School Press, 1996)


Stanford hospital clinics report card3
Stanford Hospital & Clinics Report Card GME Decision Making

  • Indicators are designed to measure SHC’s progress toward achieving its vision; other indicators are designed to measure the long term drivers of success.

  • Through the balanced scorecard, SHC :

    • Monitors its current performance (finances, resident satisfaction, learning outcomes and program process results)

    • Monitors its efforts to improve processes, educate residents

    • Enhances its ability to grow, learn and improve the quality of its fellowship and residency educational programs.


Balanced scorecard strategic perspectives
Balanced Scorecard GME Decision MakingStrategic Perspectives

Resident

How do our residents see us?

Are our programs excelling?

Institutional / Financial Growth

Mission

Vision

Strategy

Program

Processes

Are we putting our resources in the right places?

Do we continue to improve (outcomes)?

Learning


Measurement across the continuum
Measurement Across The Continuum GME Decision Making

  • PRE: Measuring events that occur before the trainee arrives

    • NRMP Results

  • PERI: During Residency

    • ACGME Survey

  • POST: After they have left training to start their career.

    • Alumni Survey


Selection of report card measures
Selection of Report Card Measures GME Decision Making

  • # Applicants/Open Positions

  • Match Depth

  • % Top Medical Schools

  • GME Internal HS Survey

  • Overall Satisfaction

  • Recommendation of Program

  • Teaching Quality

  • Curriculum Quality

  • Educational Leadership

  • Wellness Index

  • ACGME Survey

  • Compliant Responses

  • Alumni Survey

  • Faculty Eval of program

  • Resident Eval of program

  • Faculty Publications

  • Duty Hr Violations

  • ACGME Cycle Length

  • # ACGME Citations

  • Core Competency Self-Assessment

  • ITSE Scores

  • Annual Resident Publications

  • Annual Resident Presentations

  • # Safety Incident Reports

  • Specialty Board Scores

  • Core Competency Post Assessment

  • # Res in Program

  • Grants Awarded

  • Subspecialties/Program

  • Physical Space/Facilities


Voice of the residents
“Voice of the Residents” GME Decision Making


Program processes
Program Processes GME Decision Making


Program matrix
Program Matrix GME Decision Making


Learning outcomes
Learning Outcomes GME Decision Making


Financial growth
Financial / Growth GME Decision Making


Case study stanford
Case Study - Stanford GME Decision Making

  • How the DIO uses the Report Card


How do we use this data
How Do We Use this Data? GME Decision Making

  • Look at Indicators that are Resident Driven – “Voice of the Resident”

    • Is there a discrepancy between the voice of the resident and the other indicators?

      • Would the majority of the residents not choose the program again yet the Board Scores are high?


How do we use this data1
How Do We Use this Data? GME Decision Making

  • How Do the Programs Compare Against Each Other?

  • How do they compare against their ACGME Cycles?


What s next
What’s Next? GME Decision Making

  • GME Staff Brainstorming Session

  • The Why’s – Why are programs where they are?

  • Where do we need to focus our resources?


Presenting the data
Presenting the Data GME Decision Making

  • Program Directors Monthly Forum

    • Protect the Name of the Program

      • Growth and Change not Blame


Presenting the data1
Presenting the Data GME Decision Making

  • Individual Meetings with Program Directors

    • Share Complete Data


Action planning
Action Planning GME Decision Making

  • GME Staff working with Program Directors

  • Sharing Findings with GMEC and Administration


Political fallout
Political Fallout GME Decision Making

  • No Program Director wants to be at the bottom…

  • Defensiveness

  • Bragging Rights


And by the way this will help you answer
And by the way… this will help GME Decision Making you answer:

COMMON INSTITUTIONAL REVIEW DOCUMENT

Question 30b:

“Describe how the sponsoring institution monitors that each program provides effective educational experiences for residents that lead to measureable achievement of educational outcomes of the ACGME competencies.”


Questions
Questions GME Decision Making