Working with NSF: Writing, Reviewing and Rotating Tanja Pietraß

1. Working with NSF: Writing, Reviewing and Rotating Tanja Pietraß

2. The Writing Process

3. How to Write a Strong NSF Proposal • Stage 1 – Preparation • Read the GPG • Follow instructions • Start early! • Respond to review requests! • Ask a mentor for a copy of a proposal that led to an award

4. How to Write a Strong NSF Proposal • Stage 2 – Research and Writing • Read the GPG and follow the suggested format • Put your research in context to what’s been/being done • Take care when preparing your broader impact statement – it should not look like an afterthought • For CAREERs: Treat the education plan like a research project • Provide a list of ca. 5 suggested reviewers (provide address/phone/email; you may comment on relationship/expertise)

5. “… individual projects should include clearly stated goals, specific descriptions of the activities that the PI intends to do, and a plan in place to document the outputs of those activities.” GPG: A Competitive Proposal

6. How to Write a Strong NSF Proposal • Stage 3 – Review and Submission • Make it perfect – proof-read! • Have a mentor proof-read your proposal for technical input • Share the proposal with a non-expert for general input (readability, logical construction, no discipline-specific jargon, sufficient level of background) • Check your figures (legibility, numbering, referred to in text?) • Submit early! • Proof-read the final NSF-generated pdf (or better, submit a pdf)

7. The Review Process

8. The Life Cycle of an NSF Proposal Notification DGA Return Without Review Mail Panel DD Concur NSF Admin. Review Award/ Decline? Both PI has an idea! Proposal Receipt at NSF Proposal Review Notification 6 Months 30 Days 90 Days Proposal Review and Recommendation Proposal Preparation Processing and Notification Award ? Decline ?

9. The Life Cycle of an NSF Proposal 6 Months 30 Days 90 Days Proposal Review and Recommendation Proposal Preparation Processing and Notification Government Performance and Results Act of 1993 (GPRA) Target = Process 70% of proposals within 6 months of receipt at NSF

10. Administrative Review / Compliance NSF Admin. Review PI has an idea! Proposal Receipt at NSF 6 Months 30 Days 90 Days Proposal Review and Recommendation Proposal Preparation Processing and Notification

11. Administrative Review / Compliance • Proposal is checked to verify that it complies with the rules and formatting parameters delineated in the Grant Proposal Guide (GPG) • Common issues include: • On certain issues that involve sections other than the Project Description or Project Summary, the PI may be asked to do a proposal file update (PFU) within 5 business days of notification • Biographic sketch is too long, too many publications • Margins or font sizes are too small • References are not in the correct format • Improper supporting documents uploaded

12. Administrative Review / Compliance Return Without Review NSF Admin. Review PI has an idea! Proposal Receipt at NSF 6 Months 30 Days 90 Days Proposal Review and Recommendation Proposal Preparation Processing and Notification

13. Return Without Review The Proposal: • Does not address Broader Impacts as a separate section within the Project Summary • Has a budget line for a postdoctoral research associate but does not have a one page postdoctoral mentoring plan uploaded in the supporting information section • Does not meet an announced proposal deadline or close of window date (and time, where specified)

14. Return Without Review The Proposal: • Is inappropriate for funding by the National Science Foundation (or the CHE Division) • Is a duplicate of, or substantially similar to, a proposal already under consideration by NSF from the same submitter • Was previously reviewed and declined and has not been substantially revised.

15. Administrative Review / Compliance Please be sure to: • Read the NSF Grant Proposal Guide (GPG) carefully • Check that all of the necessary parts of the proposal are uploaded • Make sure that Broader Impacts are addressed in a separate section within the Project Summary (and Project Description) • Check that all of the figures appear as you intended in the .pdf file loaded into FastLane • Follow any special instructions in a solicitation

16. The Life Cycle of an NSF Proposal Return Without Review Mail Panel NSF Admin. Review Both PI has an idea! Proposal Receipt at NSF Proposal Review 6 Months 30 Days 90 Days Proposal Review and Recommendation Proposal Preparation Processing and Notification

17. Proposal Review • The proposal content sets the direction of the review • Proposers are invited to suggest names of persons they believe are especially well qualified to review the proposal or identify persons they would prefer not to review the proposal • Follow GPG Guidelines and include full contact information

18. Three Principles of Merit Review • All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge.

19. Three Principles of Merit Review • NSF projects, in the aggregate, should contribute more broadly to achieving societal goals. These broader impacts may be accomplished through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by, but are complementary to, the project. The project activities may be based on previously established and/or innovative methods and approaches, but in either case must be well justified.

20. Three Principles of Merit Review • Meaningful assessment and evaluation of NSF funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects. If the size of the activity is limited, evaluation of that activity in isolation is not likely to be meaningful. Thus, assessing the effectiveness of these activities may best be done at a higher, more aggregated, level than the individual project.

21. Merit Review Criteria • Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge • Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.

22. Elements of Merit Review • What is the potential for the proposed activity to: a. Advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and b. Benefit society or advance desired societal outcomes (Broader Impacts)? • To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?

23. Elements of Merit Review • Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? • How well qualified is the individual, team, or organization to conduct the proposed activities? • Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?

24. How to translate this …. The Intellectual Merit criterion encompasses the potential to advance knowledge. Translation into questions a peer reviewer would want to know: • What is it about (the research objective)? • What is already known and what will your research add (context)? • How will you do it (the technical approach)? • Can you do it (you and your facilities)? • Is it worth doing (intellectual merit and broader impact)?

25. How to translate this …. The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes. Translation into subjects a peer reviewer would want to know about: • Education and training • Providing opportunities for underrepresented groups • Improving research and education infrastructure • Dissemination of research results • Communicating the benefits of research to nonscientists

26. The Life Cycle of an NSF Proposal Notification DGA Return Without Review Mail Award ? Panel DD Concur NSF Admin. Review Award/ Decline? Both Decline ? PI has an idea! Proposal Receipt at NSF Proposal Review Notification 6 Months 30 Days 90 Days Proposal Review and Recommendation Proposal Preparation Processing and Notification

27. A competitive proposal is a good idea, well expressed, with a clear indication of methods for pursuing the idea, evaluating the findings, and making them known to all who need to know. A Competitive Proposal • Decisions are based on the results of merit review and other considerations, such as program budget

28. 250 Declined Awarded 200 150 Actions 100 50 0 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Rating Results of Merit Review

29. Conflicts of Interest (COI) • Peer Review at NSF needs to be the “gold standard” • Science ethics • Submitters to the federal government (e.g., NSF) are held to high standards • A breach of ethics can lead to • Being barred from submitting proposals • Fines • Jail • Trouble with your employer • Some ethics violations are a felony

30. Confidentiality • NSF Proposals – all information is confidential • Ideas, information, PI names, salaries • Rankings, reviews, reviewers’ names • Proposal existence is confidential to PI and institution • Declined proposal is always confidential • Award abstracts are published on public website • NSF program staff, reviewers, panelists, Committee of Visitors can see proposals on “need to know” basis

31. Confidentiality Do not reveal your participation in specific review activities (e.g., don’t list on resume) Never discuss panel activity or your reviews with anyone outside of NSF-supervised activities Remember panel rankings or your review may not lead to an award due to programmatic considerations COI identification relies on self-identification and list of collaborators

32. Conflicts of Interest (COI) - A COI is where an individual cannot judge a proposal fairly because of another relationship with that proposal. Unsure? Ask NSF • Institutional COIs • Current, recent (12 mo) employee or negotiating for employment • Recent (past 12 mo) honorarium or award • Reemployment plans, stock, visiting committee member • PIs, co-PIs, Senior Participants • PhD or postdoc advisor or student (lifetime) • Collaborator (past 48 mo), co-editor (past 24 mo) • Business partner, friend • Family • Other COIs • Reviewer expresses a bias against a scientific approach

33. How We Deal with COIs at NSF Removal - reviewer that has COI is not used Recusal - panelist asked to leave the room when the particular proposal is discussed NSF Office of General Counsel mediates and can make allowances COI forms are signed by panelists and kept on file Program Directors attend annual COI training Every division has an Ethics Official

34. How to Write a Substantive Review • Intellectual Merit • Broader Impact • Postdoc Mentoring Plan • Data Management Plan • Program Specific Review Criteria

35. Postdoc Mentoring Plan • Mandated by America COMPETES Act • Required if postdoc funds are requested • Supplementary document (less than 1 page) • Include, but not limited to: • Career counseling • Training in preparation of grant proposals, publications and presentations • Guidance on ways to improve teaching and mentoring skills • Guidance on how to effectively collaborate with researchers from diverse backgrounds and disciplinary areas • Training in responsible professional practices

36. Data Management Plan Required since Jan. 2011 supplementary document; less than 2 pages describes how the proposal will conform to NSF policy on the dissemination and sharing of research results Can’t submit without! See NSF webpage for FAQs Specific guidance for CHE: http://www.nsf.gov/bfa/dias/policy/dmpdocs/che.pdf

37. Data Management Plan the types of data, samples, physical collections, software, curriculum materials, and other materials to be produced in the course of the project; the standards to be used for data and metadata format and content (where existing standards are absent or deemed inadequate, this should be documented along with any proposed solutions or remedies); policies for access and sharing including provisions for appropriate protection of privacy, confidentiality, security, intellectual property, or other rights or requirements; policies and provisions for re-use, re-distribution, and the production of derivatives; and plans for archiving data, samples, and other research products, and for preservation of access to them.

38. Be a Responsible Reviewer • We seek merit review by experts and generalists • Comments are more important than the box checked (E, V, G, F, P) • Be constructive • What’s good? What’s important? What’s missing? What needs improvement? • Strengths & weaknesses • Review the proposal, not the person (except for qualifications and productivity) • Comment on BOTH standard review criteria and any additional programmatic criteria

39. Be a Responsible Reviewer… • OK to comment on clarity, budget, etc. • Avoid ad hominem and identifying comments. • Respond in a timely manner • Respond quickly even if the answer is “no” • Please suggest alternatives • Most relevant reviewers are asked first • Your fast assistance assures timely response to the proposer (6-month GPRA – Government Performance and Results Act – target) • NSF requires 3 reviews

40. Notification: Declinations

41. Declinations…. The other story • Of course, not all stories have happy endings… • Most proposals are declined • Funding rates in CHE are 20-25% • So chances are, at some point, you will receive that dreaded email… • Remember, this happens to nearly everyone, experienced investigator and novice applicant alike…

42. Please keep in mind that… • A declination is not a judgment of your value as a scientist • It is not a judgment about your or your merit! • It is a merely a judgment regarding one particular proposal in the context of • the other proposals in the pool • available funding • and possibly other issues

43. Nevertheless…. You wish to increase your competitive edge Some ideas on how to do this are the subject of this presentation

44. What not to do when you are declined It is not a good idea to call up your program director right after you learn of your declination It is difficult to have a productive discussion until you have had a chance to read and think over your reviews (including panel summary and PO comments) and perhaps discuss them with your colleagues. It is really never a good idea to call up and holler at your program director.

45. What also not to do when you are declined • It is an extremely bad idea to enlist colleagues, deans, provosts and others to lobby your program director on your behalf. • Actively discourage such behavior on the part of zealous colleagues. • Remember, we cannot and we will not discuss your proposal with anyone but you! • It is also inappropriate to lobby for your own proposal!

46. Also.. • Please do not try to guess who the reviewers are • 99.9% of the time you will be just plain wrong • In light of this, be careful when you specify reviewers you do not want us to use…. • base this on objective knowledge • you do not want to run the risk of losing a supportive reviewer

47. What to do when you are declined Email your program director and set up a time for a telephone appointment to discuss your proposal and reviews. Include your proposal number in the request! During this appointment, make every effort to listen with an objective mind, whether you agree or not. Do not waste time during this appointment arguing with your program director Do ask questions!!

48. Learn to read your reviews objectively (for everyone!) • The responses to reviews I hear from many declined PIs fall into two categories: • How could you decline me, the reviews were glowing!! • Reviewers A, B, and C are obviously incompetent idiots • who clearly did not read my proposal • who are out to get me • who know nothing about the field

49. Try to read reviews objectively For this, it is often useful to enlist a colleague, one who will “tell it like it is” Try to analyze your reviews as if they were reviews of someone else’s proposal Make a list of “to do” items that emerge from the reviews If you think a reviewer misunderstood you, try to figure out if you can make your point more clearly

50. Typical reviewer comments (to help you analyze your reviews) The topic: old hat, crowded field, not important to chemistry The vision: poorly motivated, not clear where project is going, not clear how pieces hang together, what questions are being asked? The prospects for success: not convinced it will work, needs preliminary results, not enough detail for me to evaluate, methods not up to the task, too ambitious, PI does not realize difficulties, no plan B