Using Peer Review to Evaluate the Societal Relevance of Humanities Research
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
In light of growing calls to demonstrate the societal value of academic work, this paper explores whether peer review is a reliable method to evaluate the societal relevance of humanities research. It also offers an estimate of how relevant published journal articles and books from five humanities fields are to society. By modeling two evaluation tasks involving many raters and documents, we estimate how various reviewer characteristics (such as their chauvinism and strictness) and document characteristics (such as their field and content) affect societal relevance ratings. We then compare the influence of both reviewer and document characteristics on these ratings and provide an estimate of the societal relevance of humanities research where the factors contributing to peer review unreliability are filtered out. Our results suggest that, even according to humanities scholars themselves, a substantial portion of published humanities research is not relevant to society at large. Furthermore, our results also suggest that when using peer review to decide whether a particular piece of research is societally relevant, the selection of reviewers plays a more significant role than the content of the research.
