Update of CSR Diversity Initiatives

1. Update of CSR Diversity Initiatives ACD Diversity Workgroup Subcommittee on Peer Review Early Career Reviewer (ECR) Program Monica A. Basco, Ph.D. Scientific Review Officer Coordinator, ECR Program Exec Sec, Subcommittee on Peer Review

2. Ginther et al. (2011) Findings • African American applicants were 10 percentage points less likely to receive NIH research funding compared to Whites • Suggested explanations: • Bias in peer review • Deficits in applicants’ grant writing abilities • Applications with strong priority scores were equally likely to be funded regardless of race • This suggests that problems occur at the peer review stage or earlier

3. NIH Response ACD Working Group on Diversity in the Biomedical Research Workforce met throughout 2012and developed recommendations • Created a permanent Diversity Working Group • Established a Subcommittee on Peer Review to: • Examine all hypotheses, including the role of unconscious bias, related to disparities in research awards at NIH. • Provide advice on potential interventions to ensure the fairness of the peer review system. • Eight scholars with expertise in social science, unconscious bias, stereotyping, and faculty development

4. America COMPETES Challenges • Developed in collaboration with the Subcommittee on Peer Review • Contest format with cash prizes • Solicit ideas from scientific community and other stakeholders • Two stage judging process • Technical Evaluation by Subcommittee Members • Selection of Winners by Federal Employees • $10,000 First Prize • $5,000 Second Prize

5. Challenge #1: New Methods to Detect Bias in Peer Review • How to detect bias among reviewers due to gender, race/ethnicity, institutional affiliation, area of science, and/or amount of research experience of applicants. • Best empirically based idea should: • Be theoretically based and/or hypothesis driven • Propose an experimental design • Be well-grounded in peer reviewed empirical literature • Propose measurement methods • Be implementable • Be related to the Peer Review Process

6. Challenge #1: New Methods to Detect Bias in Peer Review • Most Creative Submission should: • Propose novel concepts or translate existing concepts in a novel way • Challenge existing paradigms • Have potential to be translated for use in an experimental design • Describe creative ways to apply ideas • Be implementable • Relate to the Peer Review Process

7. Challenge #2: Strategies to Strengthen Fairness and Impartiality in Peer Review Best Submission should: • Demonstrate general knowledge of peer review practices • Be grounded in the empirical literature • Be implementable • Have potential to be delivered in a variety of formats • Demonstrate understanding of the training literature • Move theory to practice • Provide evidence that supports the effectiveness of the approach

8. Detection of Bias in Reviewer Comments • Text Analysis of Unedited Reviewer Critiques • Word Count Methods • Lexicon of terms • Achievement (awards, honors) • Ability (skill, ability) • Research (productivity, experiment) • Standout adjectives (exceptional, outstanding) • Grant evaluation words (groundbreaking, meritorious)

9. Text Analysis • Task 1: Validation of Lexicon • Summary Statements • Applicant Gender differences • Applicant Race differences • Task 2: Test of reviewer Comments by Applicant Race • Test original Lexicon • Revise and Reanalyze – Iterative process • Task 3: After accumulation of critiques • Test of Reviewer and Applicant Race differences in evaluation of grant applications

10. Anonymization Experiments – Basic Assumptions • If Racial Bias in Grant Reviews Exists: • Reviewers are aware of applicant race/ethnicity • This knowledge influences their ratings of applications • Anonymized applications from Black applicants will get better scores

11. Anonymizing Experiments – Design Options

12. Anonymizing Experiments

13. Anonymizing Experiments

14. Re-Review of ARRA Grants

15. Bias Awareness

16. Bias Awareness Questionnaire The following section includes examples of comments that could be made during review meetings. Please review the following and indicate how often you have heard similar comments in your merit review panels: Never Occasionally Very frequently 1 2 3 4 5 PI’s ethnicity: “Okay, I’m the lead reviewer for the proposal from [Name of Institution]. I can’t even begin pronounce the PI’s name, just look at number .......” Gender: e.g. “The PI is a talented young woman. She does a lot of committee work and spends a lot of time mentoring her students. It is not clear that she is making research a priority.” Faculty Rank: e.g. “The PI is a new assistant professor and the Co-PI is an associate professor. Shouldn’t the PI be concentrating on getting tenure?” Disciplinary Differences: e.g. “I don’t understand the way these educational psychologists write their proposals. It needs more tables or charts for an engineer like me to be able to understand what they are trying to say.” Reputation:e.g.“I would have expected more out of this PI; their work is always great, even though this proposal is less than their usual effort, I think we should give it a chance.”

17. Mentioning the name of the applicant Mentioning the title of the application Noting that the applicant had recently completed her K01 training Noting that several of her publications included her mentor as a co-author Noting that she has created her own research niche Voicing concern about her independence Bringing up the recent article on incarceration among Black male study participants Dr. Lakeisha Tubman, is a New Investigator from a Historically Black College. In the discussion of her application entitled, “Neuroscience of Engagement of Black Males in Hypertension Management,” a reviewer noted that she has recently completed her K01 training and several of her publications included her mentor as a co-author. While she has created her own research niche, there was some concern voiced about her independence. Another reviewer pointed to a recent article on how incarceration among Black male research participants causes problems with attrition and made the point that these findings raised concerns about the feasibility of the proposed work.

18. Mentioning that she is from a high research intensive institution Noting that the applicant has a gap in her publication history Noting that she has prior NIH funding The Chair’s joke about her knowledge of the subject matter Making the committee aware of her recent pregnancy Defense of her two year gap given the nature of her research Discussion of the high quality work coming from her department At a recent study section meeting, an application submitted by Dr. Joanna Fulbright entitled, “Epidemiology of Breast Cancer and Advanced Maternal Age,” was given an average preliminary score of 2. Although she is from a high research intensive university and has had prior NIH funding, her biosketch showed a 2 year gap in her recent publication history. The Chair, who had seen the pregnant applicant present her work at a recent conference, made a good hearted joke about the applicant knowing her subject matter inside and out. During the discussion of her application, a reviewer commented that a two year gap was not unusual given the time it takes to complete epidemiological work. Another reviewer noted the high quality work coming out of the applicant’s department.

19. Intervention Development and Testing Early Career Reviewer Program (ECR) • 3134 ECRs have been accepted into the program • 1069 have served on at least one study section • 30 percent of those who have served are from under-represented groups • Early Career Reviewer Application and Vetting System (EAVS) • ECR video created and disseminated • Outreach webinars for R15 schools