How evaluators construct assessment rationales for non-traditionally submitted research outputs

Despite the centrality of the journal article to traditional assessments of research quality and excellence, valuable research is disseminated through a variety of forms. Non-traditionally submitted outputs (NTOs) represent scholarly work that does not conform to conventional academic formats but nevertheless embodies rigorous research processes that generate new insights accessible to relevant audiences and open to scholarly critique and debate. Although epistemically legitimate, NTOs are frequently evaluated using criteria designed around assumptions that the academic article is the primary locus of scholarly value. Traditional evaluation criteria serve as key mechanisms through which peer reviewers structure assessments and through which research governance actors legitimise evaluation outcomes and promote transparency. However, these criteria may be poorly suited to capturing the distinctive merits of NTOs, potentially leading to unfair or incomplete assessments of research excellence. This study employs a ‘think aloud’ experimental design to investigate how Evaluators apply the traditional criteria of Originality, Rigour and Significance, when assessing NTOs. The analysis explores how the merits of NTOs are mapped onto existing criteria and examines the adequacy of those criteria for evaluating non-traditional forms of research dissemination. Findings show that Evaluators frequently struggle to operationalise traditional criteria when assessing NTOs, often requesting additional contextual information or attempting to translate NTO attributes into article-based evaluative frameworks. These findings suggest that without clearer, community-wide approaches to evaluating NTOs, research systems risk systematically undervaluing important forms of scholarship and limiting their contribution to wider public and scholarly debate.