peer review challenges raised by omb s draft guidelines n.
Skip this Video
Loading SlideShow in 5 Seconds..
Peer Review: Challenges Raised by OMB’s Draft Guidelines PowerPoint Presentation
Download Presentation
Peer Review: Challenges Raised by OMB’s Draft Guidelines

Loading in 2 Seconds...

play fullscreen
1 / 12

Peer Review: Challenges Raised by OMB’s Draft Guidelines - PowerPoint PPT Presentation

  • Uploaded on

Peer Review: Challenges Raised by OMB’s Draft Guidelines. Leslie Hushka, Ph.D., DABT September 30, 2003. ExxonMobil Biomedical Sciences, Inc. What Guideline Elements Are Addressed. Bulletin Goals Information Covered / Not Covered Selecting Peer Reviewers Peer Review Process

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Peer Review: Challenges Raised by OMB’s Draft Guidelines' - lara-wilson

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
peer review challenges raised by omb s draft guidelines

Peer Review:Challenges Raised by OMB’s Draft Guidelines

Leslie Hushka, Ph.D., DABT

September 30, 2003

what guideline elements are addressed
What Guideline Elements Are Addressed
  • Bulletin Goals
  • Information Covered / Not Covered
  • Selecting Peer Reviewers
  • Peer Review Process
  • Information Access and Public Comments
  • Updating Agency Guidelines
  • Summary
what is not covered in this section
Dr. Richard Belzer, President Regulatory Checkbook

Scholarly peer review  Government peer review (including ownership, objectives, selection, procedures, interests, accountability, etc)

No evidence that peer review is an appropriate remedy for the "problem" government peer review is supposed to solve

Too much attention is devoted to conflicts of interest and not enough to coincidence of interest

Dr. Steven Lewis, Distinguished Scientific Associate, EMBSI

Peer Review  Stakeholder Dialogue

Conflict of Interest  Bias

Peer Review  Sound science

What is Not Covered in This Section

SRA Forum on Peer Review - May 29 2002

bulletin goals
Bulletin Goals

... Ensure that agencies conduct peer reviews of the most important scientific & technical information

... Peer reviews are reliable, independent and transparent


  • Are these the proper goals? Are these aligned with info quality?
  • What is reliable?
  • Independent? From Agency only? Transparent how & when?
  • What criteria will be used to judge success in meeting goals?
information that is covered not covered by guidelines
Information that is Covered / Not Covered by Guidelines

...Regulatory information means any scientific or technical study that is relevant to regulatory policy . . . used by regulatory bodies

... Peer review undertaken by a scientific journal may begenerally be presumed to be adequate


  • Does journal peer review meet the amended PR - IQG standards?
  • What if the Agency re-analyzes, summarizes, interpretes original findings?
  • Does establishing a rebuttable presumption for published articles free agencies from requirement to substantiate the quality of information?
selecting peer reviewers
Selecting Peer Reviewers

... Selected on the basis of necessary scientific and technical expertise

... Broad a range of expertise as is necessary

... Do not possess real or perceived conflicts on interest


  • Will this create additional burden or discourage qualified scientists?
  • Is a broad range of expertise really necessary? Does it dilute skills & decrease rigor of review?
  • Conflicts of interest and bias - Does the 'appearance' of either of these affect selection of reviewers?
  • How will the agency measure 'perceived' conflicts of interest?
peer review process
Peer Review Process

...Select peer review mechanism based on the novelty and complexity of the science to be reviewed, the benefits and cost implications, and any ’controversy’ regarding the science


  • Are 'controversies' on the science a valid criteria for selecting a peer review method? Is ‘uncertainty’ a more appropriate term?
  • Agency must provide reviewers with information on those potential sources of controversy . . .What and how?
  • If Agency uses other firm(s) to conduct review, does it establish some dependence upon the Agency?
information access and public comments
Information Access and Public Comments

...Provide an opportunity for other interested agencies and persons to submit comments ... provided to peer reviewers

....Disclose names, qualifications of peer reviewers

...Include a certification explaining how agency has complied


  • How will the Agency address public & peer review comments on information products that have not been 'disseminated'?
  • How will agencies set up an administrative records for information products that are years away from use in rulemakings?
  • What criteria will OMB use to review ‘certifications’ for PR/IQG? Should these criteria be the same to evaluate if a review is successful? Can criteria be outcome-based and not output-focused?
updated agency guidelines
Updated Agency Guidelines

...Supplement or amend IQGs to incorporate requirements

...Guidelines for... entanglements that preclude an individuals

... Confidentiality in peer review


  • Performance standards or detailed processes? Is this appropriate? How will performance be measured?
  • Should separate agency guidelines / standards be developed by each agency?
  • Any criteria for excluding peer reviewers?
  • What, if anything, is confidential in the peer review process?
data quality contains multiple elements
Data “Quality” Contains Multiple Elements


  • Present info in accurate, clear, complete and unbiased manner
  • Identify sources, supporting data and models
  • Ensure transparency, reproducibility, and use “best available science”


  • Agency must consider how the public uses the information and ensure appropriate transparency for those uses


  • Ensure that information is not compromised through corruption or falsification

Many questions for agencies to address in developing or modifying peer review programs

Overriding questions:

  • Will peer review improve the quality of information disseminated?
  • If so, will peer review also improve the underlying quality of science used for regulatory purposes?
  • If not, what will?
  • Are there other models that will better fit these needs?