1 / 27

  • Uploaded on
  • Presentation posted in: General

NO CONFLICT OF INTEREST TO DECLARE. Can peer review improve quality?. David Smith. Peer review origin?. Ethics of the Physician Ishaq bin Ali al-Rahwi (854–931) of Syria.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

No conflict of interest to declare


Can peer review improve quality

Can peer review improve quality?

David Smith

Peer review origin

Peer review origin?

Ethics of the Physician

Ishaq bin Ali al-Rahwi (854–931) of Syria

No conflict of interest to declare

All Quality Assurance programmes seem to rely ultimately on peer review, often in the form of an expert panel, to set the standards and the limits of acceptability.

Problems with peer review

Problems with peer review

  • Provision of

    • objective standards

    • structured processes

    • training reviewers

  • Perverse incentives/perverse activities

  • Reviewer bias

  • Authority of the reviewers

  • Indemnity/protection

Gmc procedures to assess the performance of doctors who may be seriously deficient

GMC procedures to assess the performance of doctors who may be seriously deficient

  • peer review of the doctor’s practice at the workplace

  • tests of competence and skills.

    Peer reviews are conducted by three trained assessors, two from the same speciality as the doctor being assessed, with one lay assessor and include:

    • the doctor’s portfolio of training, experience, practice and self assessment

    • a review of the doctor’s medical records

    • discussion of cases selected from these records

    • observation of consultations

    • a tour of the doctor’s workplace

    • interviews with at least 12 third parties (five nominated by the doctor)

    • structured interviews with the doctor.

Failure of peer review

Failure of peer review

Given the magnitude of the resources devoted to quality assurance and the centrality of peer assessment to these efforts, the results of this literature analysis indicate the need for a global reexamination of the peer review process.

Several of the proposals discussed herein appear to have considerable potential for improving the reliability of peer judgments.

Research directed at evaluating this potential should be part of the overall reassessment of peer review.

Goldman RL JAMA. 1992;267:958-960

Reviewer agreement

Reviewer agreement

physician agreement regarding quality of care is only slightly better than the level expected by chance.

Goldman RL JAMA. 1992;267:958-960

Goldman suggestions

Goldman suggestions

  • Use of multiple reviewers

  • More objective assessments

  • Higher standards for peer reviewers

  • Use of practice guidelines

  • Use of outcome assessments

Watching the doctor watchers

Watching the doctor watchers

The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through

  • improved reviewer selection and training

  • a structured review method

  • more physician reviewers per record

Rubin et al JAMA 1992;267:2349-2354

Review of 339 hospitals pr system

Review of 339 hospitals PR system

  • Only 24% of respondents said it was very likely to improve quality while 33% said it was unlikely

  • Needs standardisation,

  • more governance,

  • bigger case load for reviews (>1% of admissions),

  • more personnel (already 1.1wte per 100 beds)

    The Process of Peer Review in U.S. Hospitals

    Marc T. Edwards & Evan M. Benjamin

    JCOM 2009 Vol. 16;10 461-467

Reviewer agreement1

Reviewer agreement

  • 13 board-certified physicians completed a total of 741 structured implicit record reviews of 95 records for patients who experienced severe in hospital adverse events while in the hospital (hypokalemia, hyperkalemia, renal failure, hyponatremia, and digoxin toxicity).

  • They independently assessed the degree to which each adverse event was caused by medical care and the quality of the care leading up to the adverse event.

  • Working in pairs, they then discussed differences of opinion, clarified factual discrepancies, and re-rated the record. The authors compared the reliability of each measure before and after discussion, and between and within pairs of reviewers.

  • The reliability of the assessment of whether the complication was iatrogenic was 0.46 before discussion and 0.71 after discussion between paired reviewers, indicating considerably improved agreement between the members of a pair.

  • However, across reviewer pairs, the reviewer reliability was 0.36 before discussion and 0.40 after discussion.

  • Similarly, for the rating of overall quality of care, reliability of physician review went from 0.35 before discussion to 0.58 after discussion as assessed by pair. However, across pairs the reliability increased only from 0.14 to 0.17

    Conclusions. When 2 physicians discuss a record that they are reviewing, it substantially improves the agreement between those 2 physicians. However, this improvement is illusory, as discussion does not improve the overall reliability as assessed by examining the reliability between physicians who were part of different discussions.

    Hoper TP et al Medical Care 2000, 38:2 152-161



Surrogate markers of quality

May be ill founded

May be biased

Open to interpretation

Unintended consequences

Limit development

Are guidelines following guidelines a review of 279 guidelines 1985 1997

Are guidelines following guidelines?A review of 279 guidelines 1985-1997

Guidelines published in the peer-reviewed medical literature during the past decade do not adhere well to established methodological standards. While all areas of guideline development need improvement, greatest improvement is needed in the identification, evaluation, and synthesis of the scientific evidence.

JAMA. 1999;281(20):1900-1905

Practice guidelines developed by specialty societies the need for a critical appraisal

Practice guidelines developed by specialty societies: the need for a critical appraisal

431 guidelines were eligible for the study. Most did not meet the criteria

Despite improvement over time, the quality of practice guidelines developed by specialty societies is unsatisfactory.

Explicit methodological criteria for the production of guidelines shared among public agencies, scientific societies, and patients' associations need to be set up.

Lancet 2000:355;103-106

Scientific evidence underlying the acc aha clinical practice guidelines

Scientific Evidence Underlying the ACC/AHA Clinical Practice Guidelines

Recommendations issued in current ACC/AHA clinical practice guidelines are largely developed from lower levels of evidence or expert opinion.

The proportion of recommendations for which there is no conclusive evidence is also growing.

These findings highlight the need to improve the process of writing guidelines and to expand the evidence base from which clinical practice guidelines are derived

JAMA. 2009;301(8):831-841

Risk adjusted outcomes funnel plots

Risk adjusted outcomesfunnel plots

Bcis peer review and angioplasty quality

BCIS ‘peer review’ and angioplasty quality

Historical ‘no holds barred’ case review is an exemplar of peer review but to be rigorous it needs a systematic format and a written record

No conflict of interest to declare

A prospective study of colonoscopy practice in the UK today: are we adequately prepared for national colorectal cancer screening tomorrow?

C J A Bowles

GUT 2004 53 277-283

Can peer review improve quality1

Can peer review improve quality?

Increasingly robust risk adjusted outcome data

Local systematic egalitarian case review arrangements

Bcis peer review

BCIS Peer Review

  • What about BCIS per review

  • BCIS were asked to set up a PR system in 1999 and felt individual PR was not feasible and designed a unit PR system.The first draft had many of the features that Goldman suggests but fell at the first few hurdles of reviewer acceptability, resources, lack of indemnity

  • At that point it was highjacked by the department of Health in order to limit the potential proliferation of of low grade angioplasty units. It has continued in this format of a box ticking exercise for new units measured against current BCIS guidelines.

  • limited as it is does it improve quality? One of the few but always contentious, surrogate measures of quality that are recorded is the number of procedures done per year.

  • Yet there are some units consistently doing less than 400 for up to 7 years

Numbers of small volume units over last ten years

Numbers of small volume units over last ten years

Number of pcis performed in 2006

2006 data: Ludman

Number of PCIs performed in 2006

Centres performing < 200 procedures

PCI program start date

No conflict of interest to declare

  • The purpose of peer review is to provide an expert view of an individual’s ability to meet professionally acceptable standards and thus help ensure an acceptable level of continued professional competence. Such review might necessarily identify individuals who do not conform to acceptable standards and who may therefore be required to undergo retraining or be suspended. If the outcome of a peer review might put at risk the ability of an individual to continue to practice, then such a review has to be rigorous in the extreme. It would need to be supported by a large volume of accurate statistics and involve comparison to a detailed set of performance standards that is accepted by the speciality as a whole

No conflict of interest to declare

  • Audit

  • Case review

  • Direct observation of clinical activity/procedures

  • 360 Appraisal

  • Login