1 / 15

Archived File

Archived File. The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files. Initiatives to Enhance Peer Review: Two-Stage Peer Review

tilly
Download Presentation

Archived File

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.

  2. Initiatives to Enhance Peer Review: Two-Stage Peer Review Anita Miller Sostek, Ph.D. Peer Review Advisory Committee August 27, 2007 National Institutes of HealthU.S. Department of Health and Human Services

  3. Problem to be Solved • Review of complex, multidisciplinary and translational research that: • Emphasizes overall significance and impact • Provides specific expertise in a wide range of scientific areas • Makes efficient use of reviewer’s time and effort • Preserves the dynamic of reviewer discussion • The challenge of reviewing complex applications has been consistently noted at each of the CSR Open Houses

  4. Two-Stage Peer Review Pilot • A two-stage peer review is similar to a scientific journal editorial board review • The first stage review of methodological and technical aspects of the science • The second stage emphasizes impact and significance • The second stage could be conducted in a variety of meeting formats (face-to-face, telephone, AED, or VED)

  5. What the Two Stages Involve • First stage involves many reviewers submitting independent assessments of scientific approach • methodological and technical merit • written critiques similar to “mail” reviews • Second stage involves a small group of experienced reviewers with broad scientific expertise • focus on impact and significance • examine all applications to ensure quality and consistency of assessments • Final scoring is done by the second stage reviewers • First stage reviewers may provide advice about scoring (perhaps A,B, C, D)

  6. What is the History? • Examples of complex and non-traditional reviews at the NIH: • Pioneer awards • SCOR (Women’s Health) Centers Review • NHLBI

  7. Where Will the Pilot be Applied? • Research Grants – concentrate on multidisciplinary and translational areas of science • Three or four IRGs that focus on disease processes or physiological systems • How to define? • Status: Under discussion • Pilot for Bioengineering Research Partnerships • Complex technology applications • Status: Currently underway • Pilot For Small Business Applications • Status: Under discussion

  8. Issues to be Addressed • Logistics of recruitment and optimal review loads for reviewers at both stages of review • Defining conflicts of interest and dyads • Scoring procedures • Advice from first stage; final scores from secondary stage • How will streamlining be conducted? • Second stage based on broader issues; e.g., impact • Consideration given to scoring advice of the first stage

  9. Timing – Will it be Faster? • Ultimately, it is hoped that Two-Stage Review will be faster than traditional reviews (each reviewer examines the full application in depth) • Requirements of the Timeline • Adequate time for distribution to two groups of reviewers • Turnaround from initial stage of review to the second • Reviewers will need to be recruited and assigned before full set of applications are received. • What can we learn from other agencies?

  10. Potential Advantages • Recruitment may be facilitated because first stage reviewers are asked to evaluate specific aspects of the applications • it may be possible to recruit earlier • mail reviewers are less limited by dyad considerations • Efficiencies will be achieved as the primary reviewer (second stage) does not need to critique approach in detail and may provide a resume • Both stages may be well suited to a structured critique formats • Greater consistency since scoring will be done by a smaller number of reviewers

  11. Process Sample - BRPs Stage One: • At least two mail reviews (first stage critiques) for each application – specialized technology and medical input • Critiques at both stages will address all review criteria but will focus mainly on areas within their expertise. • Mail reviews will be prepared a month before the second stage deadline to allow time to consider them in preparing the broader critiques. • Mail reviewers will provide preliminary scores for the whole application, unless otherwise mentioned.

  12. Process Sample - continued Stage Two: • Reviewers “senior level” and/or experienced with BRPs • Will have a month to examine the applications and the first stage critiques • The second stage reviewers will write critiques that incorporate the first stage reviews • Option available to bring first stage reviewers in by phone if their perspective is needed for discussion.

  13. Challenges of Pilots • Adapting the general concept to specific types of reviews • Avoid perception of lesser value for the first stage reviewers • Defining the main focus of the reviews at each stage • Avoid perception of lesser value for the first stage reviewers • Logistics • timing • workloads and recruitment • technology adaptations needed to allow broader reading of mail reviews within IAR

  14. Models for Two-Stage Peer Reviews • Veterans Administration Merit Review ** • Department of Defense ** • Private foundations **Previous presentations discuss review at these agencies

  15. Discussion

More Related