Evaluating Interventions Designed to Increase the Uptake of Open Research

"Much investment is going into enhancing the uptake of open research, for example by providing training, or reforming researcher assessment, or enabling institutions to share and adapt good practices from elsewhere. However, evaluation of the impact of that investment is not always easy or done well. In this webinar, we outline the approach taken by the UK Reproducibility Network (UKRN) Open Research Programme (ORP), which runs from 2021 to 2027 and involves between 20 and 45 institutions, depending on the particular intervention.

The ORP provides train-the-trainer courses to support the delivery of training, works with institutions to incorporate or enhance recognition of open research in recruitment and promotion procedures, and is developing tools to help institutions share and learn from each other. The open research space is active, dynamic and international, so the ORP is making strong efforts to collaborate while focusing on meeting specific planned outcomes. We discuss the complex evaluation task that arises from this varied set of interventions and global context, the approaches being taken, and the challenges we are facing. Our evaluation aim is to establish a baseline of open research practices at partner institutions, identify changes in open research practices at those institutions and begin to identify where those changes might relate to activities of the ORP. The primary outcome is the take-up of open research practices, but also each of the ORP intervention strands has its own process evaluation. The evaluation method will focus on pre-/post- analysis as a minimum, with the aspiration to introduce comparator group(s) where possible using Bayesian analysis. At each stage of the evaluation, the central evaluation team will analyse qualitative and quantitative measurements already captured by each particular intervention project within the ORP."