How evaluators construct assessment rationales for non-traditionally submitted research outputs | Research Evaluation | Oxford Academic

Abstract:  Despite the centrality of the journal article to traditional assessments of research quality and excellence, valuable research is disseminated through a variety of forms. Non-traditionally submitted outputs (NTOs) represent scholarship that does not conform to conventional formats but nevertheless embodies rigorous research that generates new insights accessible to relevant audiences and open to scholarly critique. Although epistemically legitimate, NTOs are frequently evaluated using criteria designed around assumptions that the academic article is the primary locus of scholarly value. Traditional evaluation criteria serve as mechanisms through which peer reviewers structure assessments and through which research governance legitimises evaluation outcomes and promotes transparency. However, these criteria may be ill-suited to the distinctive merits of NTOs. This study employs a ‘think-aloud’ experimental design to investigate how Evaluators apply the criteria of Originality, Rigour and Significance, when assessing NTOs in the UK Research Excellence Framework (REF). It explores how the merits of NTOs are mapped onto existing criteria and examines the adequacy of those criteria for evaluating non-traditional forms of research dissemination. Drawing on 68 observations, findings show that Evaluators struggled to operationalise traditional criteria when assessing NTOs, often requesting additional contextual information or attempting to translate NTO attributes into article-based evaluative frameworks. These findings are specific to NTOs and do not constitute a comparative claim about the adequacy of REF criteria across output types, however they suggest that without clearer, community-wide approaches to evaluating NTOs, research systems risk systematically undervaluing important forms of scholarship and limiting their contribution to wider public and scholarly debate.