Panel Briefing

1. Panel Briefing CAREER Panel

2. CISE Organization and Core Research Programs 70% CISE Core Programs CISE Cross-Cutting Programs 30% Cross-Foundation Programs

3. CAREER Program • The Faculty Early Career Development (CAREER) Program • National Science Foundation's most prestigious awards in support of junior faculty who exemplify the role of teacher-scholars through • outstanding research • excellent education and • the integration of education and researchwithin the context of the mission of their organizations.

4. CAREER Proposal • All CAREER proposals must include • A Summary explicitly addressing intellectual merits and broader impacts • An integrated research and education plan at their core. • A description of the proposed research project covering • Background and significance • Specific objectives • Preliminary results where appropriate • Methods and procedures to be used • Anticipated results • A description of the proposed educational activities, including plans to evaluate their impact on students and other participants; • A description of how the research and educational activities are integrated with one another; and • Results of prior NSF support, if applicable.

5. Supplementary Information • Letter from the PI’s department head • Must support the proposed integrated research and education plan • Evaluated as part of the assessment of the overall merits of the proposal • Data management plan • Postdoctoral mentoring plan (if requesting support for postdocs)

6. Panel Objectives • Provideinformative feedback to the PIs • Even the best proposals can benefit from advice • The weaker the proposals are – the more crucial the feedback • Informative and constructive feedback is especially important in the case of CAREER proposals • Provide advice to NSF Program Directorsfor making award recommendations • Provide basis for decision making by the NSF Division Directorin approving (or not) funding or declination recommendations

7. Conflicts of Interest Purpose • remove or limit the influence (or appearance of influence) of ties to an institution or investigator • preserve the trust of the scientific community, Congress, and the general public • Institutional (“statutory”) • Current, previous (12 months) or possible future employment • Spousal or off-spring employment or enrollment • Received payment for participation in advisory committees, honorarium, etc. (12 months) • Personal(“regulatory”) • Thesis advisor or student relationship • Coauthor, research collaborator within past 48 months • Family member or close friend of the PI Panelists must not be present in the room when proposals are discussed with which they have a COI.

8. Confidentiality • Your participation is confidential! • OK to say you participated in an NSF panel • Granularity: IIS Division, year 2009 • Not OK to say which panel, which day! • Panel recommendations are confidential! • Do not discuss recommendations and panel proceedings, panel membership, proposals, … • Proposals contain sensitive information • Proposals contain sensitive information; not in public domain • Do not copy, distribute or quote from them • Leave them and any notes in the room after the panel • Panel makes recommendations, not decisions

9. Proposal Evaluation • NSF Evaluation Criteria: • Intellectual Merit • Broader Impacts • CAREER proposals: • Integration of Research and Education • Letter from the department chair (required) • Data Management Plan • Postdoctoral Mentoring Plan (if applicable) • Results from Prior NSF Support (if applicable) • Human Subjects Considerations (if applicable) • Facilities and Resources • Budget

10. Standard NSF Evaluation Criteria: Intellectual Merit • Importance of proposed activity • To advance knowledge and understanding • Within the field and across fields • How well conceived and organized is the proposed activity? • Soundness and feasibility of approach, evaluation, research plan • Significance of expected contributions • Creative, original, potentially transformative research • Qualifications of the investigator • Data Management Plan • Facilities and Resources • Equipment, facilities, etc • Requested Support (budget)

11. Standard NSF Evaluation Criteria: Broader Impacts • Advance discovery and understanding while promoting teaching, training, and learning • Broaden the participation of underrepresented groups • Computer Science education, computer systems workforce • Gender, ethnicity, disability, geographic, age, etc. • Enhance the infrastructure for research and education? • Disseminate results (data, software, …) to enable further research, education, technology transfer • Societal benefits

12. Integration of Research and Education • All CAREER proposals must have an integrated research and education plan at their core. • Can be at any level – K-12, undergraduate, graduate, professional practice, community at large • Examples of activities • Development of innovative curricula, pedagogy • Outreach (community, profession) • Mentoring • Broadening participation • Enquiry-based learning • Integration of a research component into undergraduate courses • Must advance the PIs own disciplinary and educational interests and goals, as well as the needs and context of his or her organization

13. CAREER: Integration of Research & Education • There is a broad range of possible activities • Innovative curriculum and pedagogy • Outreach and mentoring • Students or the general public • At any level: K-12, undergraduate, graduate • Designing new materials and practices • Activities must be integrated

14. Writing Good Reviews • Imagine yourself in the role of the PI • Write the kind of review that you would like to receive on your own proposals • Reviews should be • Informative, substantive • Non-inflammatory • Anonymous • Substantiated • Unbiased • Constructive • Avoiding bias • Beware of sources of implicit bias • Evaluate the proposal as written • Focus on strengths and weaknesses under each review criterion • Weigh the strengths and weaknesses to arrive at an overall rating

15. Writing Good Reviews • Fill out all sections • Rating (use single rating if possible; if you must use a double rating, say why: e.g., I rate the proposal VG on intellectual merits but E on broader impacts”) • Proposal summary • Intellectual Merits • Strengths & Weaknesses • Educational Activities, including Integration of Research and Education • Strengths & Weaknesses • Broader Impacts • Strengths & Weaknesses • Summary statement • Justify the rating (E, VG, G, F, P) in terms of your overall assessment of intellectual merits, broader impacts AND integration of research and education activities

16. Writing Good Reviews • You are on the panel because you expertise relevant to the review of proposals being reviewed • Do not say: “I am not qualified to review this proposal ..”. • Such comments prompt PIs to question the entire review: • Reviewer’s qualifications to conduct a proposal review • Assignment of proposals to panels • Assignment of reviewers to proposals • Avoid unsubstantiated statements • Do not say “There is nothing novel about this work” • Instead, you might say “The novelty of this work relative to the work of XYZ et al. (see <citation>) is unclear” • Avoid inflammatory language

17. Minimizing Bias in Evaluation • Implicit bias toward a group (“schemas”): • Non-conscious hypotheses/stereotypes, often about competence • Lack of critical mass leads to greater reliance on schemas • Few women and minorities in sciences • Accumulation of disadvantage • Small bias in same direction has large effect (in salary, promotion, and prestige) over time [Valian1998]

18. Grades & Recommendations • Panelist grades: E, V, G, F, P • Avoid being overly harsh (“I never give an E”) or overly generous. • Be discriminative & use the entire spectrum P .. E • Panel recommendations: • Highly Competitive (HC): Solid proposal, deserves funding. • Competitive (C): Good proposal, but some portions unconvincing. • Low Competitive (LC): PI is encouraged to revise and resubmit • Not Competitive (NC): PI is discouraged from resubmitting • Panel recommendation is based on insights gained during discussion • Funding a project with F or P rating and declining one with E rating requires explanation by PD.

19. Panel Summaries • Please follow the format in writing panel summaries. • Do not merely list reviewers’ comments. • Should reflect the whole discussion to be useful to NSF and PI • Crisp comments to help unsuccessful PIs improve their proposals for the next competition • Comments in reviews and panel summaries should be constructive, informative, non-inflammatory, and non-discriminatory • Ignore letters of endorsement except from organizations contributing funds, data/traces, use of facilities, etc.