Archived File

1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.

2. PEER REVIEW ADVISORY COMMITTEE MEETING STRUCTUREDCRITIQUES September 26, 2005 Hortencia Hornbeak, Ph.D. Associate Director for Scientific Review and Policy DEA, NIAID

3. Outline for Today’s Discussion • Multi-pronged approach to shortening review cycle • Principal drivers of structured critiques • Structured critique formats • Structured critiques-award mechanisms • Reviewer guidance • Changing reviewer behavior • Resources needed

4. Features of NIH Peer Review thatMeet the Test of Time • Management of conflict of interest (core value) • Secure appropriate expertise (core value) • Develop collective expert advice through discussion • Provide guidance/feedback to applicant

5. Multi-pronged Approach • Electronic submission of applications • Knowledge management solutions in referral/recruitment • Electronic recruitment of reviewers for SEPs • Internet assisted review (IAR) • Structured critiques • Abbreviated summary statements • Council approval independent of scheduled meetings

6. Structured Critiques: Principal Drivers • Accelerated review for AIDS initiatives (mid 1980s) • Hyper-accelerated review of Innovation grants for HIV/AIDS hyper-accelerated review • Large increase in funds ($1.5 B) for biodefense research (2003) • Five review cycles (73 days per cycle) per FY • 60 days from receipt to review

7. Structured Critiques: Principal Drivers (cont.) • Hyper-accelerated review of Bioshield initiatives- Project Bioshield Legislation 2004 • No FACA rules • Outside IMPAC II • Constraints on FY funding • Lag time to hire and train staff

8. Five Review Cycles: FY03-04 • 73 days per cycle for most reviews • SRA team work • Flexible SRA assignments • Advance electronic recruitment of reviewers for SEPs based on letters of intent • Structured critiques • IAR assembly of abbreviated summary statements • Administrative review report to GMB and Program

9. Five Review Cycles: FY03-04 (cont.) • Lessons learned • Secure appropriate expertise with large numbers of potential reviewers in conflict • Tools developed to manage large reviews • Structured critiques • Practical guidance for reviewers • Administrative workbooks • Reviewer Support Site • Publication search macro for conflicts

10. Five Review Cycles: FY03-04 (cont.) • Effective flexible working teams • Conflict of interest management • Development of staff training resources • Effective communications with stakeholders • Pre-meeting teleconferences to educate reviewers • Increase use of teleconferences for reviews

11. STRUCTURED CRITIQUE FORMAT

12. Structured Critique - Benefits Goal • Improve utility for stakeholders by: • More focused discussions • More concise written evaluations • Facilitated preparation of abbreviated summary statements

13. Structured Critique - Challenges • Requires change in behavior of reviewers • May adversely affect “tutorial” aspect • May not capture adequately complexity and/or subtleties • May lead to applicant appeal

14. Structured Critiques: Simple and Complex Mechanisms • R03 • R21 • R01/U01 • P54 Regional Centers - two-tiered review • P01/U19 • UC6 Biocontainment laboratories

15. Structured Critique Template • Tailored for each initiative • Initiative-specific review criteria • Use short phrases to describe each review criteria • Based on the five NIH review criteria as a scaffold for initiative specific criteria

16. Critique Structured Format • Significance: • Strength # 1 [strongest](1-2 lines) • Justification/explanation (2-4 lines) • Strength # 2 (1-2 lines) • Justification/explanation (2-4 lines) • Strength # X (1-2 lines) • Justification/explanation (2-4 lines)

17. Critique Structured Format (cont.) • Weakness # 1 [most serious weakness] (1-2 lines) • Justification/explanation (2-4 lines) • Weakness # 2 (1-2 lines) • Justification/explanation (2-4 lines) • Weakness # X (1-2 lines) • Justification/explanation (2-4 lines)

18. Critique Structured Format (cont.) • Continue with format in previous slides for: • Approach • Innovation • Investigator • Environment Criteria

19. EXAMPLE OF A STRUCTURED CRITIQUE

20. Example of Structured Critique RESUME AND SUMMARY OF DISCUSSION:  This  (adjectival rating) application entitled “" was submitted by  (organization),  Principal Investigator. The applicants propose to  (one sentence  summary of specific aim(s). The principal strengths identified by reviewers include , , , and . The major weaknesses are , , , and . After discussion, the panel concluded that ,  and scored the application in the range.

21. Example of Structured Critique (cont.) OVERALL EVALUATION This is an excellent application in response to the RFA…... It is based on solid hypotheses and preliminary evidence that XX inhibitors can protect from toxin-mediated pathogenicity of B. anthracis. The team is highly experienced and has worked together to establish the background for the project. Goals are clearly laid out, experiments are generally well described, and there is a high probability of significant advances in therapy of anthrax-related toxicity in patients.

22. Example of Structured Critique (cont.) CRITIQUE • Significance • Strength 1 - novel approach to block enzymatic conversion of a bacterial XX protoxin in several bacteria, including B. anthracis • Strength 2 - excellent likelihood of successful development of lead compounds • Strength 3 - excellent opportunity to test lead compounds directly in toxin-treated animals • Weakness 1 - unknown toxicity related to furin inhibitors

23. Example of Structured Critique (cont.) • Approach • Strength 1 - solid rationale and preliminary evidence for Pseudomonas-related toxin XX, including protection of toxin-induced death in mice • Strength 2 - lead compound XXX has potent binding to target • Strength 3 - excellent team to conduct studies

24. Example of Structured Critique (cont.) • Approach (cont.) • Weakness 1 - Is furin block in vivo sufficient to cure infection or only protect from XX-mediated toxicity? • Weakness 2 - in preliminary results, the PI concludes that “XXX actually enters cells”, but the data only demonstrate that “biotinylated XX” enters cells - they are quite different substances. To further understand the apparent lack of cytotoxicity of XXX, permeabilized cells should be used.

25. Example of Structured Critique (cont.) • Innovation • Strength 1 - detailed mechanistic study of XXX effect on XX toxicity to RAW cells • Strength 2 - study of kinetics and time course of protection from XX and cell-surface bound iodinated XX • Strength 3 - design of XX analogs including synthetic peptides, polyamines and related peptidomimetics • Strength 4 - in vivo studies examining efficacy of XXX on TNF production and on XX induced toxicity in animal models • Weakness 1 –the approach not innovative and could benefit from collaborative expertise for these studies.

26. Example of Structured Critique (cont.) • Investigator The PI is experienced in peptide hormone studies and peptide processing. Collaborator Dr. XX is experienced in peptide synthesis. The team has successfully worked together in the discovery of polyarginine furin inhibitors. The application would be strengthen by additional collaborative expertise for XX studies • Environment Lab facilities of the PI are adequate for the studies. Presumably those of Dr. XX’s institution are also adequate.

27. Example of Structured Critique (cont.) • Budget Appropriate for the level of effort proposed. • Animal Welfare Comments: A letter of approval of an animal protocol was included. It does not indicate if both mice and rats are approved for the study. The PI does not address the “five questions” about animal usage. Nature of anesthesia is not indicated.

28. STRUCTURED CRITIQUES: MECHANISM-SPECIFIC EXAMPLES

29. Structured Critiques: Simple to Complex Initiatives • R21- Rapid Response Grant Program on Bioterrorism-Related Research • U01/U19 RFA Cooperative Research for the Development of Vaccines, Adjuvants, Therapeutics, Immunotherapeutics, and Diagnostics for Biodefense • Biodefense Countermeasure Development

30. Structured Critiques: Benefits – All Mechanisms • More focused assessments of the applications • Increased time for discussion of competitive applications • Enabled committee to manage larger number of applications per SEP. • Faster SS completion for awarded grants

31. Structured Critiques - R21 • RFA Rapid Response Grant Program on Bioterrorism-Related Research (2002) • MechanismR21 • Number of applications 303 • Review committees 3 SEPs • Number of members/SEP 25 • Streamlined applications 33 percent

32. Structured Critiques - R21 (cont.) • Electronic critique templates • 3 SRAs assigned • Resume and summary of discussion templates • Faster SS completion – 4 weeks

33. Structured Critiques - U01s and U19s • RFA Cooperative Research for the Development of Vaccines, Adjuvants, Therapeutics, Immunotherapeutics, and Diagnostics for Biodefense (2003) • MechanismU01, U19 • Number of applications 186 • Review committees 3 SEPs • Number of members/SEP 40 • Streamlined applications 33 percent • Electronic critique templates

34. Structured Critiques - U01s and U19s (cont.) • Resume and summary of discussion templates • Templates for resume and summary of discussion • 3 SRAs assigned • Faster SS completion - 3 weeks

35. Structure Critiques - Project Bioshield • Biodefense Countermeasure Development Hyper-accelerated Review (2004) • MechanismN/A (small grants) • Number of applications 66 • Letter of intent/program Pre-approval • Review committees 1 SEP • Number of members/SEP 23 • Streamlined applications 60 percent

36. Structure Critique - Project Bioshield (cont.) • Teleconference review • Receipt to award 60 day • Electronic critique templates • Enabled committee to manage the large workload in a hyper-accelerated time frame • Faster SS completion - 2 weeks Not standard summary statements

37. Guidance Provided to Reviewers • Orientation teleconference with members from multiple committee for standardization • Electronic critique template tailored to the initiative. • Instructions for Reviewer Manual tailored for each initiative or SEP • Overlapping reviewers among the review panels

38. Changing Behavior of Reviewer • Clearly stated expectations • Focused rational rationale and objectives • Required buy-in from all stake holders • NIH • NIAID • reviewers • Make it easy across the board

39. Resources Needed to Decrease Time • Sufficient numbers of trained staff • Effective SRA/GC Work teams • Flexible staff assignments • Versatile and state of the art IT support

40. Summary • Preserve features that make NIH Peer Review work with any changes adapted • Manage conflict of interest (core value) • Secure appropriate expertise (core value) • Develop collective expert advice through discussion • Provide guidance/feedback to applicant • Structured critiques are applicable to many award mechanisms: simple to complex

41. Summary (Cont.) • Reviewers must be properly oriented in pre-review teleconferences • Reviewers do adapt to change • Structured critiques decrease time for summary statement preparation • Resulting product (SS) more useful to stakeholders

42. Summary (cont.) • Structured critiques along with other innovations can shorten the review cycle • Electronic submission of applications • Knowledge management solutions in referral • Electronic recruitment of reviewers for SEPs • Internet assisted review (IAR) • Abbreviated summary statements • Council approval independent of scheduled meetings

43. Questions?