Automating ERA benchmarks, and a call for expressions of interest from institutions - COKI

"COKI team member Julian Tonti-Filippini led the construction of a pilot Research Impact Evaluation System (RIES) demonstrating the feasibility of conducting an on-demand, ERA-like analysis for research-active institutions (locally and globally), using journal-level metadata from the Australian Research Council and article-level metadata from publicly available datasets.

Excellence in Research for Australia (ERA) is a periodic assessment that is conducted by the Australian Research Council (ARC). The assessment focuses on the activity of 42 Australian higher education providers (HEPs) across 236 ANZSRC fields of research (FoR). Performance is assessed (per HEP and FoR) by comparing research outputs to local and world benchmarks. Analysis has a citation-focus and draws from publication metadata provided by the participating HEPs.

An ERA report is usually compiled for release every three to five years and employs citation-focused methodology in the analysis of research output data, self-reported by the participating institutions. However, on August 26 this year, Education Minister Jason Clare put ERA 2023 on hold due to the significant reporting burden the process imposes on the sector. There has long been an interest in automating parts of this process to reduce this burden, with a 2021 consultation on the ERA process also noting an interest in enhancing transparency regarding the construction of benchmarks and performance measures. 

The Curtin Open Knowledge Initiative (COKI) aggregates bibliometric and bibliographic data from publicly available sources such as Crossref, Unpaywall, OpenCitations, Microsoft Academic Graph, and OpenAlex. The resultant BigQuery database contains metadata for over 120 million research publications and forms the foundation for further analysis by the COKI team.

We developed RIES to demonstrate how the COKI database may be used to run an ERA-like analysis. The methodology is guided by published ERA methods and makes use of journal-level metadata from the ERA 2023 Journal List. The workflows can be extended, to include any institution (with a ROR identifier) and any research-topic vocabulary that has been assigned to research articles (eg, via machine-learning classifiers). This flexibility, combined with the on-demand capabilities of the system will help us to model and test the approaches proposed for new national research assessments."