1 / 15

Archived File

Archived File. The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files. PILOT TWO-STAGE PEER REVIEW 2008-2009 Don Schneider.

kalia-briggs
Download Presentation

Archived File

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.

  2. PILOT TWO-STAGE PEER REVIEW 2008-2009 Don Schneider June 8,2009 National Institutes of HealthU.S. Department of Health and Human Services

  3. Remember a golden age of peer review? • 100 applications per study section • 20 regular members, no ad hoc or temporary members • 18 assignments each, 12 written and 6 reading • Interactive discussions

  4. Changes in 50 years of peer review • Increase transparency – even share summary statements with applicants • Increase focus – research, from faculty avocation to prime focus • Increase importance – research, major role in nation’s economy and leadership • Increase in scientific complexity/diversity • Decrease in reviewer willingness to commit time to peer review

  5. Two-stage peer review addresses breadth and depth • Two-three stage 1 reviewers per application, write critiques (depth) • Three stage 2 reviewers per application (breadth) • All final decisions by stage 2 members – interactive discussions • Each application examined by at least 5 reviewers

  6. CSR has conducted pilots • Small business grant applications (last 3 cycles) • Transformative R01 applications (RFA-RM-08-029) in progress • Challenge grant applications (RFA-OD-09-003, Recovery Act) in progress

  7. 17-Week cycle makes for a tight timeline Week post-receipt date Activity 0 Receipt date 2-3 Apps to SRO 4-6 Apps to stage 1 6-9 Reviews in IAR 9-13 StSx meetings 11-17 Summaries out

  8. Stage 1 Reviewers • 1-5 applications each • 2-3 reviewers per applications • Spot on scientific expertise • Reviewers write full critiques, independently • Non-FACA, only suggest scores

  9. Stage 2 Reviewers • Three reviewers per application • Each reviewer assigned to at least 10 applications, typically 15 • One of three writes draft resume • Broad perspective • FACA, responsible for all scoring

  10. Review outcomes show promise • Discussions were thoughtful • Reviewers and staff liked outcomes • Survey of reviewers was positive: would like own applications reviewed this way, stage 1 and 2!

  11. Pilots show advantages and disadvantages • High satisfaction by stage 1 and stage 2 reviewers, would prefer for own applications • More work for review staff (more recruiting, shorter time frames) • Some sense of isolation/non-involvement by stage 1 reviewers, although several stage 2 who could not attend meeting agreed to be stage 1

  12. An Aside: Prebuttal was piloted last cycle • After posting stage 1 critiques, draft summaries were sent to applicants who had one week to answer “errors” in review • Error rate is estimated to be 3%; however, about 80% of applicants responded • Prebuttals were provided to stage 2 reviewers along with critiques • Majority of stage 2 reviewers (73%) found prebuttals helped evaluations • Prebuttals were included in summary statements – plan to poll program

  13. Editorial Board platform may have niche • When science requires multiple experts, or when application numbers are high • When science is complex/translational • When opportunity is single shot (RFA) • Science is reviewed by real experts (spot on) • Broad editors ensure perspective, assessment of impact, and better score calibration

  14. Acknowledgement: Volunteers for two-stage pilot • Lawrence Boerboom • Alexandra Ainsztein • David Balasundaram • Fouad El-Zaatari • Karin Helmers • Neelakanta Ravindranath • Andrea Kopstein – survey • Richard Panniers – IMPAC wizard

  15. Look to a new age of peer review? • X 100 applications per study section • X 100 stage-1 reviewers, 3 critiques per application • 20 stage-2 members • X 15 assignments each • Interactive discussions

More Related