A Psychometric Analysis and Primer for Decision Making in Public Third-Party Funding Allocation

A significant portion of research is financed by public resources, with the expectation that it will deliver societal benefits in return. But how good are current funding allocation processes at effectively identifying and supporting research ideas that will end up contributing to societal good? In this primer, we address the critical challenge of assessing the effectiveness of funding allocation procedures by treating the selection process as a diagnostic procedure and applying psychometric tools to its assessment. We highlight an apparent disconnect between the mission of funding beneficial research and prevailing practices in funding allocation, emphasizing a lack of understanding regarding how to measure and predict the quality of research proposals. We advocate for greater transparency in funding decision processes, and for empirical investigations of how funding systems may yield optimal outcomes under existing constraints and objectives.