Educational outcomes measurement what we ve learned what s ahead l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 41

Educational Outcomes Measurement: What We’ve Learned, What’s Ahead PowerPoint PPT Presentation


  • 135 Views
  • Uploaded on
  • Presentation posted in: General

Educational Outcomes Measurement: What We’ve Learned, What’s Ahead. Derek T. Dietze, MA President.

Download Presentation

Educational Outcomes Measurement: What We’ve Learned, What’s Ahead

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Educational outcomes measurement what we ve learned what s ahead l.jpg

Educational Outcomes Measurement:What We’ve Learned, What’s Ahead

Derek T. Dietze, MAPresident

Presentation given at the 2006 Alliance for CME Annual Meeting as: Executive Director of CME Veritas Institute for Medical Education, Inc. Hasbrouck Heights, New Jersey


Session objectives l.jpg

Session Objectives

You will be better able to:

  • Evaluate you organization’s educational outcomes measurement (EOM) in the context of others’ efforts

  • Identify opportunities to advance EOM

  • Describe the potential impact of EOM on the future of CME


Who is in the audience today l.jpg

Who Is in the Audience Today?

  • What is your affiliation?

  • Medical education/communication company (MECC)

  • Hospital

  • Medical school

  • Medical specialty society

  • Health system

  • Commercial interest

  • Other


Who is in the audience today4 l.jpg

Who Is in the Audience Today?

2. I have been involved in CME for _____

1 = less than 1 year

2 = 1 to 3 years

3 = 3 to 5 years

4 = 5 to 10 years

5 = more than 10 years


Audience opinion poll l.jpg

Audience Opinion Poll

3. To what extent do you agree with the following statement? Compared with other organizations, my organization is doing a great job of educational outcomes measurement.

1 = Strongly disagree

2

3

4

5

6 = Strongly agree

7 = Not applicable


Audience opinion poll6 l.jpg

Audience Opinion Poll

4. To what extent do you agree with the following statement?Educational outcomes measurement is a fad that will pass with time.

1 = Strongly Disagree

2

3

4

5

6 = Strongly Agree


Audience opinion poll7 l.jpg

Audience Opinion Poll

5. To what extent do you agree with the following statement? Implementing EOM is essential to the long-term success of my organization.

1 = Strongly disagree

2

3

4

5

6 = Strongly agree

7 = Not applicable


Audience opinion poll8 l.jpg

Audience Opinion Poll

6. To what extent do you agree with the following statement? I need to learn a lot more about EOM.

1 = Strongly disagree

2

3

4

5

6 = Strongly agree

7 = Not applicable


Slide9 l.jpg

CME should be a strategic asset for improving performance and enhancing patient care.

CME in the public interest.


From the ceo of the accme dec 2005 l.jpg

From the CEO of the ACCME (Dec. 2005)

“On the research front, primary studies and syntheses no longer need to ask if CE, in general, improves practice or other outcomes because there is so much evidence that many kinds and combinations can do so.”

From Robertson, Umble and Cervero, Journal for Continuing Education in the Health Professions 23, 146, 2003


Eom the past l.jpg

EOM: The Past

  • Early-mid 1990s: discussion focused on the nature and definitions of “educational outcomes”

  • Alliance for CME conference on educational outcomes

  • Early-adopters and innovators

  • Since 2000: “we need to do EOM!”


The value of cme l.jpg

The Value of CME

“A CME evaluation should focus on identifying,measuring, and describing the value provided by an intervention that leads to enhanced performance, improved healthcare quality, and reduced costs.”

Moore DE. A framework for outcomes evaluation in the continuing professional

development of physicians. In: Davis D, Barnes BE, Fox R, eds. The Continuing

Professional Development of Physicians: From Research to Practice.

Chicago, Ill: American Medical Association; 2003.


Levels of educational outcomes l.jpg

Levels of Educational Outcomes

Adapted from: Moore DE. A framework for outcomes evaluation in the continuing professional development of physicians. In: Davis D, Barnes BE, Fox R, eds. The Continuing Professional Development of Physicians: From Research to Practice. Chicago, Ill: The American Medical Association; 2003.


Measurement alliance for cme competencies survey l.jpg

Measurement: Alliance for CME Competencies Survey*

  • Competency 2.12, “…offer educational consultation that emphasizes measurement in physician organizations to identify goals for education that are specific to the practice and measurable…,”

  • Rated the lowest in terms of “responsibility” and “attainment” by CME leaders, managers, and coordinators (2.6/2.6).

*King R, North S. Alliance for CME Competency Areas Research Project Report. Alliance for CME; 2004.


Forces driving eom l.jpg

Forces Driving EOM

  • Accountability for effectiveness, impact, and use of resources

  • Desire for “exemplary” status

  • Performance improvement movement

  • MOC and MOL requirements/initiatives

  • Pay for performance movement

  • Increasingly a requirement of commercial support

  • ACCME is “raising the bar” for providers

  • CME community’s desire for continuous improvement


Performance improvement movement l.jpg

Performance Improvement Movement

American Medical Association

October 6, 2004Performance Improvement Activities approved for credit

Stage A: Learning from current practice performance assessment

Stage B: Learning from the application of PI to patient care

Stage C: Learning from the evaluation of the PI effort

May 23, 2005Internet Point of Care now eligible for credit> Review original clinical question(s).

> Identify the relevant sources from among those consulted.

> Describe the application of their findings to practice.


Performance improvement movement cont l.jpg

Performance Improvement Movement (cont.)

  • American Academy of Family Physicians: Performance Improvement in Practice


Moc and mol initiatives l.jpg

MOC and MOL Initiatives

American Board of Medical Specialties: Part IV Maintenance of Certification

“Evidence of evaluation of performance in practice”

  • July 2003: ABFM introduced new MOC requirements for family physicians.

  • Part IV requirement replaced the traditional clinical chart review. Focus on improving quality of care, rather than on record-keeping.

  • Each FP must complete 1 QI project during his or her recertification cycle to comply with Part IV.


Slide19 l.jpg

Board of Directors Action Plan Fiscal 2004

“Assure ACCME accreditation remains relevant by addressing the needs of medical boards to quantify learning and practice outcomes of individual physician learners.”


Pay for performance movement l.jpg

Pay for Performance Movement

Phase in of Medicare Pay for Performance

Q. Does the Academy (AAFP) have anything to help me see what P4P might be like?A. Yes. METRIC -- Measuring, Evaluating and Translating Research Into Care -- lets you earn 20 Prescribed CME credits in your office as you complete performance measurement and improvement projects.


Commercial interests expectations l.jpg

Commercial Interests’ Expectations

Increasingly, commercial interests are requiring CME providers to:

  • Include a sound outcomes measurement plan in grant requests

  • Become more familiar with EOM methodologies and tools

  • Report results for evaluation and future planning


Current status of eom l.jpg

Current Status of EOM

  • Increasing understanding of EOM among some CME providers and MECCs

  • Some measurement methodologies gaining acceptance and popularity

  • Few standards or best practices

  • Some measurement beyond “satisfaction” into “learning” and “performance”

  • Lack of scientific rigor

  • Much talk

  • Limited experimentation

  • Fear


Eom challenges l.jpg

EOM Challenges

  • EOM competence, skills, and resources

  • Time

  • High level support & resources

  • Practical EOM methodologies/tools

  • Funding/support for EOM from commercial interests

  • Actual/perceived lack of access to “the data”

  • “Rigor mortis” and “perfection paralysis”

  • Fear of the results


Current eom methodologies tools l.jpg

Current EOM Methodologies & Tools

  • Follow-up surveys

  • Immediate pre- and post-activity surveys/tests

  • Control group surveys

  • Focus groups

  • Clinical assertions

  • Clinical/case vignettes

  • Commitment to change

  • Pre- and post-activity prescription measurement

  • Chart audits

  • QA data


Pre post clinical assertion with control l.jpg

n=117

Single-factor ANOVA:Significant at P < 0.01

n=129

n=33

Strongly Disagree

Strongly Agree

Pre/Post Clinical Assertion with Control

“’Early aggressive therapy’ is treatment initiated within the first 6 months of RA onset.”

Data courtesy of Veritas Institute for Medical Education, Inc.


Pre post case vignette patient with as l.jpg

Pre/Post Case Vignette: Patient With AS

  • How would you treat a 28-year-old male patient exhibiting the following symptoms:

    • Morning stiffness

    • X-ray shows bilateral (sacroiliitis)

    • Swollen, painful knee

    • ESR 10

    • CBC normal


Case as patient percent of pre post discussion responses l.jpg

Case: AS PatientPercent of Pre-/Post-Discussion Responses

“How would you treat?”

NSAIDs

SSZ

MTX

Anti-TNF

Combination

Percent of Responses

Data courtesy of Veritas Institute for Medical Education, Inc.


Case as patient percent change in pre post discussion response l.jpg

Case: AS PatientPercent Change in Pre-/Post-Discussion Response

“How would you treat?”

NSAIDs

SSZ

MTX

Anti-TNF

Combination

Percent Change in Responses

Data courtesy of Veritas Institute for Medical Education, Inc.


Case vignette post activity with control l.jpg

Case Vignette: Post-activity with Control

N=46 for both groups

P=0.05

Outcomes, Inc. CME Metrics Report, July 25, 2005. Challenging Cases in Gastroenterology—Then & Now, Jobson Education Group. Satellite symposium at Digestive Disease Week, May 15, 2005.


Pre post follow up clinical assertions with control l.jpg

Live only participants do not maintain learning

Live + EM participants maintain better learning through follow up

Pre/Post/Follow-up Clinical Assertions with Control

Only patients with severe and/or refractory cases of urticaria should be managed with steroids.

Strongly agree

*ANOVA P < 0.001

Mean Rating

Strongly disagree

n=390

n=1,753

n=1,374

FU = Follow-up survey EM = Enduring material

n=335

n=229

n=67

n=372

Dietze DT, Magazine HI. Enhancing and Sustaining Learning: Report of a Comprehensive CME Initiative and Educational Outcomes Measurement Plan (forum). Alliance for CME 31st Annual Conference; January 24, 2006; San Francisco, California.


Performance improvement project l.jpg

Performance Improvement Project

Purpose:

Decrease the risk of recurrent vascular events in patients with AMI and CAD by initiating therapy with lipid-lowering agents during hospitalization.

Measurement Point:

Percentage of defined AMI and CAD patients who received a lipid lowering agent charge during stay.

Multidisciplinary Team on PI Project:

Cardiology Physicians, Telemetry Nursing Staff, Quality Services, CME Department, Pharmacy Decision Support (data specialist)

Information courtesy of Susan G. McAlexander, MPA, CHES, CME Coordinator, Providence St. Peter Hospital, Olympia, WA, February 2006.


Measurement data collection analysis l.jpg

Measurement, Data Collection & Analysis

  • Lipid use rates compared monthly and quarterly for changes after interventions.

    • Education – CME and other

    • Forms adjustments

    • In-services

    • Monthly project task force meetings to project planning, discuss interventions and results.

  • Aggregate and specific rate collection performed in the Quality Department utilizing charge data and chart review.

  • Assessment of knowledge post CME single activity intervention performed by CME department.

  • Monthly tends reported on graphs.


  • Results and application to future cme l.jpg

    Results and Application to Future CME

    • Goal percentage attained in 3rd quarter for AMI patients and 4th quarter for ICS patients.

    • Greatest change noted in the cardiology and internal medicine sections. Family medicine N too low to have statistical significance.

    • Final report to CIC included a recommendation to tie CME into future PI projects for maximal impact.

    • CME was seen to have a positive and reinforcing effect on physician learning and practice change.


    Measurement is in your future l.jpg

    Measurement Is in Your Future!


    From the ceo of the accme dec 200535 l.jpg

    From the CEO of the ACCME (Dec. 2005)

    “The Board wants to make it clear that the ACCME values an accreditation system that rewards providers

    • for establishing a mission that identifies improvements in physician performance and/or patient or population health as organizational goals, and

      b)for having a process in place to measure and increase its effectiveness in meeting this mission.”


    From the ceo of the accme dec 200536 l.jpg

    From the CEO of the ACCME (Dec. 2005)

    “Over the next few months…presentation of draft new compliance criteria and policy through which ACCME will reward providers for having processes in place:

    • To facilitate changes in knowledge, strategy, performance-in-practice, healthcare delivery or patient health.

    • To facilitate life-long learning based on individualized needs assessment where effectiveness is measured by the learning or improvements that occur.

    • To deliver education in support of the ACGME/ABMS Core Competencies, around specialty-specific curricula and in the context of the IOM Competencies.”


    Audience opinion poll37 l.jpg

    Audience Opinion Poll

    7. To what extent do you agree with the following statement? New ACCME compliance criteria for measurement of CME effectiveness would be a positive step forward for the CME community.

    1 = Strongly disagree

    2

    3

    4

    5

    6 = Strongly agree


    Audience opinion poll38 l.jpg

    Audience Opinion Poll

    8. To what extent do you agree with the following statement? Compared to other organizations, my organization is doing a great job of EOM.

    1 = Strongly disagree

    2

    3

    4

    5

    6 = Strongly agree

    7 = Not applicable


    Audience opinion poll39 l.jpg

    Audience Opinion Poll

    9. To what extent do you agree with the following statement? Implementing EOM is essential to the long-term success of my organization.

    1 = Strongly disagree

    2

    3

    4

    5

    6 = Strongly agree

    7 = Not applicable


    Vision for eom l.jpg

    EOM standards and benchmarks

    EOM best practices

    Proven methodologies and tools

    More efficient and effective CME

    Improved patient care

    Improved patient outcomes

    Improved population health

    Vision for EOM


    Thank you for your participation l.jpg

    Thank You for Your Participation!

    Derek T. Dietze, MAPresident

    [email protected]

    Queen Creek, Arizona


  • Login