the nih peer review process l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The NIH Peer Review Process PowerPoint Presentation
Download Presentation
The NIH Peer Review Process

Loading in 2 Seconds...

play fullscreen
1 / 47

The NIH Peer Review Process - PowerPoint PPT Presentation


  • 127 Views
  • Uploaded on

The NIH Peer Review Process. Sally A. Amero , Ph.D. NIH Review Policy Officer Office of Extramural Research. 2010 NIH Regional Seminars. The NIH Peer Review Process. Two-tiered Process. Mandated by law – PHS Health Act Defined in federal regulation – 42 CFR 52h

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The NIH Peer Review Process' - binh


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the nih peer review process

The NIH Peer Review Process

Sally A. Amero, Ph.D.

NIH Review Policy Officer

Office of Extramural Research

2010 NIH Regional Seminars

the nih peer review process2
The NIH Peer Review Process

Two-tiered Process

  • Mandated by law – PHS Health Act
  • Defined in federal regulation – 42 CFR 52h
  • Further defined in NIH policy
  • Per year:
    • Nearly 80,000 applications
    • Over 18,000 reviewers
the nih peer review process3
The NIH Peer Review Process
  • Initial peer review
    • Recommendations on scientific and technical merit
    • Scientific Review Groups (SRGs or “Study Sections”)
  • Advisory Council or Board
    • Recommendations to the ICs on funding, appeals, program
    • priorities
    • “Council”
  • Final funding decisions – IC Director
the nih peer review process4
The NIH Peer Review Process

Application received – CSR*

Assignments made

 

Initial peer reviewFunding considerations

SRG; study section Institutes or Centers (ICs)

IC or CSR* Duals possible

Scientific Review Officer Program Officer

 

Second level of review Funding decisions

Council or Board (IC) IC Director

Award!

*CSR = NIH Center for Scientific Review

the nih peer review process5
CSR Review

Most R01s, fellowships, and small business applications

Some Program Announcements (PAs, PARs)

Some Requests for Applications (RFAs)

Institute/Center Review

IC-specific features

Program projects

Training grants

Career development awards

RFAs

The NIH Peer Review Process

Referral to an SRG

The review locus is stated in the

Funding Opportunity Announcement.

the nih peer review process6
The NIH Peer Review Process

To Request a Scientific Review Group

  • Cover letter of application
      • Application title
      • FOA # and title
      • Request:
        • Assignment to particular SRG or study section
        • Assignment to particular IC for funding consideration
      • Disciplines involved, if multidisciplinary
      • Explanation for late application

Not all requests

can be honored.

SRG rosters are posted 30 days before the SRG meeting:

http://era.nih.gov/roster/index.cfm

http://www.csr.nih.gov/committees/rosterindex.asp

the nih peer review process7
The NIH Peer Review Process

Scientific Review Officer (SRO)

  • First level of peer review
    • Designated Federal Official
    • Extramural scientist administrator
    • Identifies and recruits reviewers
    • Manages conflicts of interest
    • Oversees arrangements for review meetings
    • Presides at review committee meetings
    • Prepares and releases summary statements
the nih peer review process8
The NIH Peer Review Process

Peer Reviewers

  • Recruitment
    • Expertise
    • Stature in field
    • Mature judgment
    • Impartiality
    • Ability to work well in a
    • group
    • Managed conflicts of
    • interest
    • Balanced representation
      • Gender
      • Geography
      • Diversity
      • Seniority
    • Availability
the nih peer review process9
The NIH Peer Review Process

Types of Scientific Review Groups (SRGs)

  • “Chartered” SRGs
    • Multiyear terms
    • Formal appointment process
    • May include temporary members for special expertise
  • Special Emphasis Panels (SEP)
    • Ad hoc membership
    • Often meet only once
the nih peer review process10
The NIH Peer Review Process

Types of Reviewers

  • Regular reviewers – permanent and temporary
    • Preliminary impact/priority scores, criterion scores,
    • written critiques
    • Final impact/priority scores
  • Other Contributing Reviewers (“mail” reviewers)
    • Written critiques, criterion scores, preliminary impact/priority scores
    • Cannot submit final impact/priority scores
the nih peer review process11
The NIH Peer Review Process

Reviewer Assignments

  • For each application:
    • ≥ Three qualified reviewers are assigned (“2 + 1”)
    • Assignments are made by the SRO
      • Based on the scientific content of application
      • Expertise of the reviewer
      • Suggestions from the PI on types of expertise –
        • not names!
      • Suggestions from Program staff
      • Suggestions from SRG members
      • Managing conflicts of interest
      • Balancing workload

Assignments are

confidential!

the nih peer review process12
The NIH Peer Review Process

Conflicts of Interest (COI)

  • COI between a reviewer and an application:
  • Financial
  • Employment
  • Personal
  • Professional
  • SRG membership
  • Other interests

Two COI vouchers are submitted

by each SRG member.

the nih peer review process13
The NIH Peer Review Process

Scientific Review Groups (SRGs)

  • Make recommendations on merit - not funding!
    • Scientific and technical merit
    • Budget and project duration
    • Protection of human subjects, inclusion plans, vertebrate
    • animals, biohazards
    • Resource Sharing Plans
    • Other administrative factors
  • Impact/priority scores
  • Criterion scores
  • Written critiques
the nih peer review process14
The NIH Peer Review Process

Confidentiality

  • All confidential materials, discussions, documents are
  • deleted, retrieved, or destroyed
  • Reviewers sent guidance with applications
  • Application information provided on secure websites or
  • protected portable devices
  • All questions must be referred to SRO
  • SRG meetings are closed to the public
  • Program staff may observe SRG meeting

Do not contact reviewers directly!

the nih peer review process15
The NIH Peer Review Process

Overall Impact: Likelihood for the project to exert a sustained, powerful influence on the research field(s) involved

  • In consideration of:
    • At least five scored criteria
      • Receive individual, numerical scores
      • Additional criteria in certain announcements
    • Additional review criteria
      • As applicable for the project proposed
      • Do not receive individual, numerical scores
      • Additional criteria in certain announcements
the nih peer review process16
The NIH Peer Review Process

See “Review Criteria at a Glance”

(http://grants.nih.gov/grants/peer/reviewer_guidelines.htm)

the nih peer review process25
The NIH Peer Review Process

NIH Scoring System

  • Numerical scores
    • 1.0 (exceptional) to 9.0 (poor)
    • Final impact/priority score - average of individual scores x 10
    • Individual criterion scores
    • Ranked by percentile for certain mechanisms
    • Not Discussed (ND) - streamlining
    • Other designations (NR, DF, AB, NP, etc.)

Final impact/priority scores

range from 10 through 90.

the nih peer review process26
The NIH Peer Review Process

NIH Scoring System

  • Preliminary scores (before the SRG meeting)
    • Entered by assigned reviewers and discussants in
    • secure website
    • Made available to other SRG members
  • Final overall impact/priority scores (at the SRG
  • meeting)
    • Voted by private ballot
    • All eligible SRG members vote

Reviewers are instructed to revise their

criterion scores after the meeting.

the nih peer review process27
The NIH Peer Review Process

Phases of Process

Score Descriptors

the nih peer review process28
The NIH Peer Review Process

Streamlining

  • Allows discussion of more meritorious applications
    • Less meritorious applications tabled at the
    • SRG meeting, designated Not Discussed (ND)
    • Requires full concurrence of the entire SRG
    • Summary statement:
      • Reviewer critiques
      • Individual criterion scores
      • No numerical, overall impact/priority score
the nih peer review process29
The NIH Peer Review Process

Streamlining

  • Score order of review
    • SRG discusses most meritorious applications first
    • Entire SRG decides when to stop, which applications will
    • not be discussed in panel
  • Other order of review (e.g., IC assignment,
  • mechanism)
    • SRO prepares a list of average preliminary scores
    • Distributes to SRG
    • Entire SRG decides which applications to discuss
the nih peer review process30
The NIH Peer Review Process

Pre-Meeting SRG Procedures

  • SRO
    • Performs administrative review of applications
    • Recruits reviewers, arranges for meeting date and site
    • Assigns 3 SRG members to each application
    • Makes applications available to reviewers
      • Internet Assisted Review (IAR) site or on CDs
      • Usually about six weeks before the SRG meeting
    • Instructs reviewers in review procedures
    • Monitors posting of initial scores and critiques in IAR

Documents for Reviewers are available at:

http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines

the nih peer review process31
The NIH Peer Review Process

Phases of Process

Structured Critiques

  • New summary statement format
    • Bulleted comments from reviewers, less text
    • Criterion scores from assigned reviewers
    • Decreases variability
    • Increases quality of information in critiques
    • More succinct, better organized
    • Encourages evaluative statements
    • Ensures that reviewers address all review criteria and considerations

Critique templates are available at:

http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines

the nih peer review process32
The NIH Peer Review Process

Phases of Process

Templates for Reviewers

Links to

definitions

of review

criteria

the nih peer review process33
The NIH Peer Review Process

Pre-Meeting SRG Procedures

  • Reviewers
    • Examine assignments
    • Submit Conflict of Interest and Confidentiality voucher
    • Read applications, prepare written critiques in templates
    • Enter preliminary scores into IAR
    • Read and consider other critiques and preliminary scores
    • Make travel and hotel arrangements

Preliminary scores and critiques may be due several days or a week in advance!

the nih peer review process34
The NIH Peer Review Process

Post-submission Materials

  • Applications submitted for Sept. 25th, 2010 and later:
    • Will only accept administrative materials resulting from
    • unanticipated events, such as
      • Revised budget page(s) (e.g., due to new funding)
      • Biographical sketches (e.g., due to the loss of an
      • investigator)
      • Letters of support or collaboration (e.g., due to the loss
      • of an investigator)
      • News of an article accepted for publication
  • Special provisions for training grants and certain FOAs
the nih peer review process35
The NIH Peer Review Process

SRG Meetings

  • Agenda
    • Call to Order - Chairperson
    • Policy and instructions - SRO
    • Discuss applications one at a time
    • Where feasible:
      • In score order
      • Cluster New Investigator (NI) applications
      • Cluster clinical applications
    • Score each application by private ballot after its discussion
    • Discuss other considerations
      • Budget
      • Resource Sharing Plans
the nih peer review process36
The NIH Peer Review Process

SRG Meeting Procedures

  • Discussion format
    • Members with conflicts excused
    • Initial levels of enthusiasm stated
    • (assigned reviewers and discussants)
    • Primary reviewer - explains project, strengths, weaknesses
    • Other assigned reviewers and discussants follow
    • Open discussion (full panel)
    • Levels of enthusiasm (assigned reviewers) re-stated
    • Individual SRG members vote
    • Other review considerations discussed (budget)
the nih peer review process37
The NIH Peer Review Process

SRG Meeting Procedures

  • If 60 applications/SRG meeting
  • ~ 50% streamlined, 30 applications to discuss and score
  • If 9 hour SRG meeting
  • ~ ½ hour introduction, streamlining
  • ~ 1 hour lunch, 2 x 15 minute breaks
  • Leaves
  • ~ 14 minutes on average/application
  • ~ 3 - 4 minutes/reviewer

Clarity and brevity are essential!

the nih peer review process38
The NIH Peer Review Process

After the Review

  • eRA Commons (http://era.nih.gov/commons/index.cfm)
    • Final Impact/Priority Score available three days after the
    • SRG meeting
    • Summary statement available 4 – 8 weeks after meeting
      • Available also to Program Officers at that time
      • Confidential document
      • Available to:
        • PD/PIs
        • NIH officials
        • Advisory Council members
the nih peer review process39
The NIH Peer Review Process

Summary Statement

  • First page
    • NIH Program Officer (upper left corner)
      • Name
      • Contact information
    • Final Impact/Priority Score or other designation
    • Percentile (if applicable)
    • Codes
      • Human subjects
      • Vertebrate animals
      • Inclusion plans
    • Budget request

A favorable score does not guarantee funding!

the nih peer review process40
The NIH Peer Review Process

Summary Statement - continued

  • Subsequent Pages
    • Description (provided by applicant)
    • Resumé and Summary of Discussion (if discussed)
    • Reviewer critiques – essentially unedited
      • Follow review criteria for mechanism
      • Protections for Human Subjects
      • Inclusion Plans
      • Vertebrate Animals
      • Biohazards
      • Budget
    • Administrative Notes
    • Meeting roster
the nih peer review process41
The NIH Peer Review Process

Appeals Process

  • NIH Program Officer = Point of Contact
    • If the outcome is unfavorable, consider your options:
      • Revise and resubmit application
        • Consider critiques in summary statement
        • Address critiques in introduction and text
      • Appeal the review outcome
        • Procedural deficiencies
        • Factual errors
        • May result in re-review of same application by
        • the same or different SRG

Discuss your options with your Program Officer!

the nih peer review process42
The NIH Peer Review Process

Appeals Process

  • Unresolved appeals are presented to the IC Council
    • Council options:
      • Support the SRG review
      • Support the appeal, recommend a re-review
        • Application could be deferred for next round
        • Application cannot be modified or updated
    • Results of a re-review cannot be appealed further
    • Council cannot overturn the SRG review or impact/priority
    • score
the nih peer review process43
The NIH Peer Review Process

Advisory Council/Board

  • Second level of review – recommendations on:
    • Research priority areas
    • Policy
    • Appeals
    • Funding
    • Quality of SRG review
  • Members
    • Scientists from the extramural research community
    • Public representatives
    • Appointed to multi-year terms
    • Appointed as Special Government Employees
the nih peer review process44
The NIH Peer Review Process

Advisory Council/Board

  • Balanced representation
    • Expertise
    • Stature in field
    • Mature judgment
    • Impartiality
    • Managed conflicts of interest
    • Balanced representation
      • Gender, Diversity
      • Geography, Seniority
the nih peer review process45
The NIH Peer Review Process

Funding Considerations

  • Authority of the IC Director
    • Scientific and technical merit (initial peer review)
    • Council recommendations
    • Relevance to program priorities in IC
    • Compliance with policies
    • Number of meritorious applications received
    • Availability of funds
    • Advice of IC Program Staff
the nih peer review process46
The NIH Peer Review Process

Additional Information

  • Enhancing Peer Review Initiative
  • http://enhancing-peer-review.nih.gov/
  • Office of Extramural Research Peer Review Process
  • http://grants.nih.gov/grants/peer_review_process.htm
  • Peer Review Policies & Practices
  • http://grants.nih.gov/grants/peer/peer.htm
  • Center for Scientific Review
  • http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/
the nih peer review process47
The NIH Peer Review Process

Contact Information

Sally Amero, Ph.D.

NIH Review Policy Officer

Extramural Research Integrity Liaison Officer

Office of Extramural Programs

Office of Extramural Research

National Institutes of Health

ameros@od.nih.gov