Associations of topic-specific peer review outcomes and institute and center award rates with funding disparities at the National Institutes of Health

  1. Michael S Lauer
  2. Jamie Doyle
  3. Joy Wang
  4. Deepshikha Roychowdhury

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This paper provides the basis for further discussion about the perceived inequities in NIH funding based on race. The strengths of this manuscript are the detailed breakdown of the available data in order to evaluate for biases, the availability of data for multiple years (2011-2015) and the consideration of alternate explanations (e.g. new applications vs resubmissions; single vs multi PI). With that said, given their conclusion that Institute (IC) assignment was the main determinant of funding rates, the approach for IC assignment should have been discussed. Other issues relate to the complexity of statistical analyses and a lack of clarity on confounding issues towards firm conclusions.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

A previous report found an association of topic choice with race-based funding disparities among R01 applications submitted to the National Institutes of Health (‘NIH’) between 2011 and 2015. Applications submitted by African American or Black (‘AAB’) Principal Investigators (‘PIs’) skewed toward a small number of topics that were less likely to be funded (or ‘awarded’). It was suggested that lower award rates may be related to topic-related biases of peer reviewers. However, the report did not account for differential funding ecologies among NIH Institutes and Centers (‘ICs’). In a re-analysis, we find that 10% of 148 topics account for 50% of applications submitted by AAB PIs. These applications on ‘AAB Preferred’ topics were funded at lower rates, but peer review outcomes were similar. The lower rate of funding for these topics was primarily due to their assignment to ICs with lower award rates, not to peer-reviewer preferences.

Article activity feed

  1. Version published to 10.7554/elife.67173 on eLife
  2. eLife

    Reviewer #4 (Public Review):

    The goal of the manuscript was to add to the research on the rates of success of African American/Black PI in their pursuit of NIH funding. The authors specifically addressed variability in funding levels of NIH Institutes and Centers(ICs). The authors were successful in identifying that there are differentials rates of award rates by IC. The authors describe that topic choice was not associated with funding after accounting for IC assignment which vary in their funding rates.

  3. eLife

    Reviewer #3 (Public Review):

    This analysis focuses on funding success for a set of NIH R01-mechanism grant applications submitted between 2011 and 2015, with a focus only on those which had white and Black Principal Investigators (PIs). It is presented as a follow-up to the previously published paper from Hoppe and colleagues in 2019, uses the same population of applications and relies on the same analysis of application text to cluster these applications by topic. The authors set out to determine how success rates associated with the application's proposed topic may be determined by the success rates associated with the Institute or Center within the NIH to which the application had been assigned for potential funding. This is a critical and important investigation that is of high potential impact. The scholarship of the Introduction and Discussion, however, fails to convey this to the reader. There are many recent publications in the academic literature that address why a disparity of funding to AA/B investigators, and a disparity of funding of topics that are of interest to AA/B investigators, are such critical matters for the NIH to identify and redress. Similarly, the Discussion and Conclusions sections do not suggest any specific actions that may be recommended by these findings, which is an unfortunate oversight that limits the likely impact of this work.

    The significance of this work is limited by a number of methodological choices that are unexplained or have not been justified and therefore appear to be somewhat arbitrary. While it can be necessary to draw category lines in an investigation of this type, it is necessary to provide some indication of what would happen to the support for the central conclusions if other choices had been made. This includes the exclusion of multi-PI applications if the Black PI was not the contact PI, the definition of AA/B-preferred ICs as the top quartile (particularly given the distribution of success rates within this quartile), the definition of AA/B-preferred topics as the 15 word clusters that accounted for only half of the AA/B applications, and the ensuing inclusion of only 27% of the AA/B applications. Arbitrary choices to use only a subset of the data raise questions about what the conclusions would be if the entire dataset of grants assigned across all of the ICs, and on all of the topics, was used.

    A fundamental limitation to this manuscript is that the authors are relying on an indirect logic of analysis instead of simply reporting the success rates for applications with AA/B and white PIs within each IC. The primary outcome deployed in support of the central conclusion is a reduction of the regression coefficient for the contribution of PI race to award success and an elimination of statistically significant contribution of research topic preferred by AA/B applicants to the award success once IC success was partialed out. The former analysis is interpreted in imprecise terms instead of simply reporting what magnitude of effect on the white/Black success rate gap is being described. And the latter analysis appears to show a continued significant effect of PI race on award success even when the IC success rate is included. The much more intuitive question of whether award rates for white and AA/B applicants differ within each IC has not been addressed with direct data but the probit model outcome suggests it is still significantly different. This gives the impression that the authors have conducted an unnecessarily complex analysis and thereby missed the forest for the trees- i.e. even when accounting for IC award rates there is still a significant influence of PI race.

    The manuscript is further limited by atheism omission of any discussion of how and why a given grant is assigned to a particular IC (this is exacerbated by incorrect phrasing suggesting the applicant "submits an application to" a specific IC) and any discussion of the amount of the NIH budget that is assigned to a given IC and how that impacts the success rate. This is, at the least, necessary explanatory context for the investigation.

  4. eLife

    Reviewer #2 (Public Review):

    The paper by Lauer et al provides further insight into the factors that might determine why RO1 applications from AAB (African American Black) principal investigators appear to fare worse than their white counterparts. Their work is derived from an earlier analysis published by Hoppe et al that found 3 factors determined funding success among AAB PIs. These included decision to discuss at study section, impact score, and topic choice. The latter, topic choice (community and population studies) appeared to represent more than 20% of the variability in funding gaps. This raised the question of whether there was reviewer bias at study sections. In the Lauer paper, after controlling for several of these variables, the authors found that the topic choice of AABs (ie. preferred topics) were indeed important in respect to funding, but they uncovered the fact that the topic choices occurred more frequent in ICs that had lower funding rates. Thus the authors conclude that the disparity between AAB and white investigator RO1s is very dependent on topic choice which ultimately ends up in larger ICs with lower funding percentiles.

    Overall the paper is relatively straightforward and could be important as It provides some additional data to consider. It is in fact basically a re-analysis of the Hoppe paper, but that is reasonable since that paper left many unanswered questions. Its implications however are less clear, and these raise additional questions of importance to the extramural scientific community as well as IC leadership.

    Overall the reader is left with the unsettling question: Can we just wish away these disparities based on IC funding rates? (Figure 1).

    1. Why would topic choice of community engagement or population studies fare worse at an Institute such as AI rather than at GM if both have the relatively same proportion of preferred topics, and both have relatively high budgets compared to other institutes. Is there one or more ICs that drive the correlations between IC funding and preferred topics or PIs?

    2. Since only 2% of all PIs are AAB does that represents another issue of low frequency relative to the larger cohort?

    3. It would be valuable to know if community engagement or population studies in total do worse than mechanistic studies. The authors do admit that preferred topics of AABs in general fare worse(Figure 2, Panel B).

    4. Another concern is that the data are up to 2015; it has now been five years and things have changed dramatically at NIH and in society. There are now many more multiple PI applications including AABs that may not be the contact PI yet are likely to be in a preferred topic area.

    5. There is nothing in the discussion about potential resolutions to this very timely issue; In other words we now know that the disparity in funding is such that AAB RO1s do worse than white PIs because they are selecting topics that end up at institutes with lower funding rates. Should the institutes devote a set aside for these topic choices to balance the portfolio of the IC and equal the playing field for AABs? Are there other alternative approaches?

  5. eLife

    Reviewer #1 (Public Review):

    This manuscript by Lauer et al follows up on previous articles that ask the question whether there are funding disparities at the National Institutes of Health for African American or Black (AAB) investigators. The investigators breakdown the analysis by race, topic of proposal, and NIH institute-Center (IC) to which an application was assigned. They conclude that the most important factor in determining funding is the Institute assignment with lower funding rates related to the funding capacity of a particular Institute (e.g National Eye Institute vs Minority Health and Health Disparities). The present study is a welcome addition to this debate since if biases do exist, NIH needs to address these. The strengths of this manuscript are the detailed breakdown of the available data in order to evaluate for biases, the availability of data for multiple years (2011-2015) and the consideration of alternate explanations (e.g new applications vs resubmissions; single vs multi PI, etc). A weakness of the data is that if their conclusion is that Institute assignment was the main determinant of funding rates, why wasn't the approach for Institute assignment discussed? Are there possible biases in this assignment besides keyword searches? There is also the question of whether there is circular logic operating here. The Minority Health and Health Disparities received the most AAB applications but had one of the lowest funding rates. Wouldn't this Institute be expected to be one in which AAB applicants would try to direct their application to? This manuscript is sure to generate additional discussion on this topic which is an important step in trying to address the issue of potential funding disparities. However as the authors point out the fact that only 2% of the applications submitted to the NIH were from AAB investigators is of concern.

  6. eLife

    Evaluation Summary:

    This paper provides the basis for further discussion about the perceived inequities in NIH funding based on race. The strengths of this manuscript are the detailed breakdown of the available data in order to evaluate for biases, the availability of data for multiple years (2011-2015) and the consideration of alternate explanations (e.g. new applications vs resubmissions; single vs multi PI). With that said, given their conclusion that Institute (IC) assignment was the main determinant of funding rates, the approach for IC assignment should have been discussed. Other issues relate to the complexity of statistical analyses and a lack of clarity on confounding issues towards firm conclusions.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)

  7. Version published to 10.1101/2020.12.27.424490v2 on bioRxiv
  8. Version published to 10.1101/2020.12.27.424490v1 on bioRxiv