Responsible Research Assessment: OPERAS activities and initiatives
OPERAS 2024-04-17
Initiated by the San Francisco Declaration on Research Assessment (DORA) in 2012, the movement for research assessment reform has been steadily advancing for over a decade. This push has been characterised by a series of initiatives, each building on the last, to reshape how research is evaluated. The Coalition for Advancing Research Assessment (CoARA) represents the latest effort, culminating in the creation of the Agreement on Reforming Research Assessment (ARRA). This agreement marks a significant step in the ongoing evolution of research evaluation practices and reflects a growing consensus on the need for evaluation metrics that not only measure research outcomes but also encourage ethical practices and acknowledge diverse contributions to the scientific community.
OPERAS is taking part in the transformative change of the research culture landscape through its dedicated participation in various activities focused on Responsible Research Assessment (RRA). Below you will find a summary and highlights from the work OPERAS is doing within the EU Funded project – GraspOS and its participation in the CoARA working groups.
GraspOS
A fundamental project that contributes to the EOSC agenda, GraspOS aims to develop, assess and operate an open and trusted federated infrastructure for next generation research metrics and indicators, offering data, tools, services and guidance. Next generation metrics is one of the eight dimensions identified by the League of European Research Universities (LERU) in their roadmap for cultural change and how Open Science interacts with Universities. And crucially about the project, the development of the criteria should be open science-aware. Thus, openness and transparency are fundamental values in GraspOS.
For the federated infrastructure to be developed effectively in various contexts, the project is deploying nine pilots. Each pilot focuses on a particular community. OPERAS is leading the thematic pilot that focuses on creating generic research assessment criteria for the Social Sciences and Humanities (SSH).
This role aligns perfectly with OPERAS vision and mission which extends towards realising Open Science in SSH research, aspiring to create a scholarly communication system that benefits researchers, academics, students, and the wider society globally, by removing barriers to knowledge access.
The pilot, therefore, has community engagement as a priority when designing pathways towards research reform. So far, OPERAS has consulted the SSH community in the two initiatives described below:
Community of Practice
In January 2024, alongside with partners from the GraspOS Community of Practice, we held a discussion on the role Open Science can play in fostering a more inclusive research assessment process that prioritises quality over quantity in the Social Sciences and Humanities. Outcomes of the discussion can be found here.
In summary, participants pointed out that there are challenges related to assessing scientific rigour of non-traditional research outputs. Another obstacle was also mentioned: how to formally recognise the diverse roles of the activities of SSH Researchers? One particular example could be the role of the translator. This issue extends to a wider problem of multilingualism in scholarly communication, a topic which is currently being researched from various angles and forums, including the CoARA Working Group on multilingualism, where OPERAS participates. Identifying and including new roles within the CReDiT taxonomy system would be a significant step forward.
Consultation Workshops
OPERAS built on the work from the Community of Practice by designing and delivering three consultation workshops. The workshops are an outcome of the commitment of OPERAS as a pilot within GraspOS to identify the needs of the community in terms of what should be included in Responsible Research Assessment (RRA). The key aspect here is the open science-aware approach of the project. Prior to the delivery of the workshops, OPERAS engaged with representatives from the HuMetricsHSS initiative to discuss experiences and lessons learned from this pivotal effort in the research evaluation area. While the workshops for GraspOS did not follow the protocol set out by Humetricshss, its philosophy and principles were a source of inspiration and are perfectly aligned with the SSH pilot and OPERAS.
The online workshops were delivered during March, April 2024. In total, 18 participants from 11 countries took part, and discussed the following topics:
Determining core values and priorities for SSH in the context of research assessment
Balancing quantitative and qualitative criteria in research assessments.
Discuss the diversity of research outputs (e.g. books) and activities (e.g. peer review) relevant to SSH for research assessment purposes
Ensuring the responsible application of various indicators.
Exploring the role of digital platforms, tools, and services in supporting research assessment for SSH scholars.
The analyses of the outcomes of the workshops will be shared and communicated in the coming months. Preliminary results will be shared at the OPERAS Conference and the presentation by Fotis Mystakopoulos on day 2, titled: Contextualising Research Assessment for Social Sciences and Humanities: Insights and Challenges from the GraspOS Project Pilot.
CoARA Working Groups
By joining CoARA, OPERAS has committed to advancing the coalition’s principles through active participation in working groups focused on addressing multilingualism and language biases, as well as recognizing and rewarding peer review processes. 
OPERAS’s engagement is a continuation of its ongoing initiatives, such as its Special Interest Group (SIG) on Multilingualism and the development of platforms like GoTriple, which promote multilingual practices. Additionally, OPERAS’ service Peer Review Information Service for Monographs (PRISM) aligns with its support for transparency in peer-review practices, further underscoring its dedication to reforming research evaluation in a manner that is both equitable and progressive. Below you can see a quick overview of how OPERAS is participating in the two working groups:
Multilingualism and Language Biases in Research Assessment
Within this working group, various activities focus on the current landscape and state of multilingualism, working progressively towards identifying best practices in recognising multilingualism and language-aware assessments. OPERAS participates in the wider activities of the working group and actively engages in Task Force 3, concentrating on providing policy advice and developing a toolbox for language-aware assessments.
Recognising and Rewarding Peer Review
This working group is concerned with one of the most important scholarly activities of researchers which is Peer Review. The fundamental question is how can this activity be defined, recognised and rewarded responsibly in a research assessment exercise? The group is actively working towards producing recommendations for three categories of peer review: i) at the article/preprint level, ii) for books/monographs and, iii) for funding proposals (grants). OPERAS is particularly involved in examining the peer review process for Books/Monographs, exploring, for instance, the potential utilisation of PRISM in research assessment
OPERAS Conference 2024
This year’s theme is Opening Collaboration for Community-Driven Scholarly Communication. On day two, the OPERAS Conference will have a distinct flavour of research assessment through two dedicated sessions. One focuses more on open access publishing, ebooks and peer review, where the other includes updates from the GraspOS project as well as presentations from the activities of CoARA and the working groups OPERAS participates in. On day three, the Advocacy SIG will deliver a workshop on RRA for the SSH focusing on initiatives such as DORA, CoARA, the Hong Kong principles for assessing researchers, and the Helsinki Initiative on Multilingualism in Scholarly Communication. Their role in the RRA movement so far, and how different stakeholders can interact with the various initiatives will be discussed.
We hope to see everyone in Zadar and debate the very important topic of research assessment reform!
Note on Contributors:
Author: Fotis Mystakopoulos
Reviewer: Carol Delmazo