Home ArtsHums OPERAS Advocating for Responsible Research Assessment: highlights from the workshop by the OPERAS Advocacy SIG

Advocating for Responsible Research Assessment: highlights from the workshop by the OPERAS Advocacy SIG

OPERAS 2024-08-01

Blog post created by the OPERAS Advocacy SIG. Authors in alphabetical order: Avanço, Karla; Ekanger, Aysa; Ernst, Elisabeth; Koukounidou, Sylvia; Mystakopoulos, Fotis & Świetlik, Marta.

1. Introduction

The OPERAS Advocacy Special Interest Group currently focuses on a different topic each year. From September 2023 to August 2024, the group concentrated its efforts on Responsible Research Assessment (RRA). One outcome of this work was the workshop “Advocacy for Responsible Research Assessment for the Social Sciences and Humanities” held in Zadar during the OPERAS conference. We, participants of the Advocacy SIG, present in this blog post the methodology used to design the workshop and the results achieved.

2. About responsible research assessment

Research assessment is often seen as the cause of many problems in the current research environment. An assessment or evaluation occurs in multiple contexts and with different purposes: to appraise an individual for hiring or promotion; to analyse different project proposals and decide which one will receive funding; and to rank institutions, which is somewhat controversial (de Rijcke et al., 2023). 

It has been recognised that these activities rely heavily on quantitative metrics. There has also been an argument that quantitative measures do not give the full picture when it comes to evaluating the quality of a research output or its impact. It is also worth noting that moving away from quantitative measures, or limiting them at least, has been challenging (Tregoning, 2018). These issues have led to the movement of Responsible Research Assessment which is an “umbrella term for approaches to assessment which incentivise, reflect and reward the plural characteristics of high-quality research, in support of diverse and inclusive research cultures” (Curry et al., 2020, p. 7). The movement also places an emphasis on “sensitivity to local and particular contexts” (Curry et al., 2020, p. 7), a concept really important to the Social Sciences and Humanities (SSH). Curry et al. (2020, 7-16pp.), then, presents several initiatives and efforts as “movers and shapers”.

3. Preparations for the workshop

The first step of the workshop preparation consisted of looking at the initiatives listed by  Curry et al. (2020) and choosing the ones that should be included in this activity. We decided to focus on a few key ones, including some more recent initiatives: DORA, CoARA, the Helsinki Initiative, and the Hong Kong Principles for assessing researchers. The rationale was simple: we settled on DORA and CoARA as the key declarations in the movement, with DORA being the initial one, and CoARA the latest one. The Helsinki initiative has a distinct flavour with a focus on multilingualism which is also key to SSH and finally, the Hong Kong Principles are about research integrity which are important for all domains and disciplines.

After having decided which initiatives would be the object of the workshop, the Advocacy SIG established the goals to be achieved:

  • Participants get an overview of the RRA initiatives we chose;
  • Participants get something practical out of it, such as possible actions that they could (help) implement at their organisations that would be contributing to RRA.

Participants should work in groups, and various ways of grouping them were considered: according to participants’ interests, by initiative (they personally are interested in, or that their organisation has signed), by the type of stakeholder they are or want to address, as personas in a roleplay scenario, by challenges (as presented in the CoARA toolbox), or by their level of knowledge about initiatives.

We decided to group the participants by the type of stakeholder they represent, facilitated by the fact that the registration form for the OPERAS conference invited registrants to choose a stakeholder group already. We chose the following six stakeholder types:

  • Research performing organisations
  • Researchers
  • Libraries
  • Publishers
  • Research infrastructures
  • Policy makers
SIG members that organised the workshop

As many people in the OPERAS community take on more than one stakeholder role, it would not be a problem for anyone to find a group where they would fit.

The following step consisted of exploring the initiatives from the standpoint of these stakeholders. The different initiatives explicitly address specific stakeholders, but their recommendations are relevant for more groups. We divided stakeholder groups between themselves and looked at which recommendations from the four chosen initiatives were relevant for each stakeholder group. The result of this exercise can be seen in the Zenodo Materials.

This exercise was replicated at the workshop, as it would allow the participants to gain an overview of the initiatives in a more engaged way (instead of, for example, listening to a presentation about the initiatives, or looking at posters). It was set up as a warm-up exercise.

For the main exercise, we aimed at an activity that would encourage workshop participants to engage in detailed discussions. However, to achieve the objective of generating various actions relevant to the participants contexts, the scope of the exercise should be  narrower. Each group should pick one recommendation from the sorted statements to develop it into actionable ideas. 

At the end of the workshop, we introduced two more initiatives to the participants. They were: the HumetricsHSS initiative and the SCOPE Evaluation Framework. While all four declarations and initiatives initially chosen summarise and promote the idea of responsible research assessment, HumetricsHSS and SCOPE are frameworks for workshops to be run at different venues, institutions or departments. Therefore, the workshop concludes not only with ideas on how statements could be implemented, but with tangible recommendations on the most relevant workshop frameworks. Those two frameworks could be replicated like-for-like by participants, or be used as inspiration to establish their own workshops.

4. Outline of the workshop

We designed the workshop to foster active participation, minimising presentations and maximising engagement over a structured 90-minute period, as follows:

  • Welcome and intro of the Advocacy SIG, 5 min 
  • Presentation about the responsible research assessment landscape and the aims of the workshop (incl Q&A), 15 min 
  • Group exercise 1, 15 min
  • Group exercise 2, 30 min
  • Presentation of group results for exercise 2 (max 2 minutes per group), 20 min 
  • Wrap-up of the workshop and a presentation about SCOPE and HumetricsHSS, 5 min

The session began with a brief welcome and introduction to the Advocacy SIG by Karla Avanço, followed by Elisabeth Ernst’s overview of the responsible research assessment landscape and workshop goals (including Q&A).

As participants entered, they chose their stakeholder groups from “chair islands” labelled accordingly. This setup facilitated targeted discussions. Despite a 15-minute delay due to an extended coffee break, shorter-than-planned introductory presentations helped us regain the schedule by the start of the second group exercise.

In the first group exercise led by Aysa Ekanger, participants sorted statements by relevance for their chosen stakeholder groups. These statements were derived from various research assessment initiatives (ARRA, DORA, Helsinki, Hong Kong Principles). Each group received a set of statements on small pieces of paper and clustered them based on their significance to the stakeholder group they represented.

The second exercise, led by Marta Świetlik, was more intensive. Each group selected one recommendation from the sorted statements to develop into actionable ideas. Using a detailed “Concept Sheet,” participants outlined their plans, focusing on several key sections:

  • Chosen statement (recommendation) and stakeholder group: This section required participants to clearly identify the specific recommendation they were addressing and the stakeholder group it pertained to. The choice of statement was critical as it set the direction for the entire exercise, ensuring that the subsequent steps were relevant and focused.
  • Idea title (optional): Participants were encouraged to craft a catchy and memorable title that conveyed the essence of their idea. This title was meant to encapsulate the core message and appeal of the concept, making it easily recognizable and engaging for others.
  • Short description: This section required an outline of the main challenges the group aimed to address with their recommendation. Participants had to describe the specific issues or gaps they identified and articulate their approach to tackling these challenges. This description was intended to provide a clear and concise overview of the proposed solution and its significance.
  • Allies: In this part, participants identified the necessary partners and existing allies who could support the implementation of their idea. They considered specific individuals, social groups, or institutions that had the influence, resources, or expertise needed to turn the concept into action. This section emphasised the importance of collaboration and strategic partnerships.
  • Vision for implementing the idea: This detailed section required participants to describe the actions needed to put their idea into practice. They had to consider existing or obtainable resources and identify any specific competencies required for successful implementation. This section aimed to transform the conceptual recommendation into a practical and actionable plan.
  • Risks: Finally, participants detailed potential pitfalls and strategies for mitigation. They identified possible challenges or obstacles that could hinder the implementation of their idea and proposed ways to address these risks. This section was crucial for ensuring that the plan was robust and realistic, with contingencies in place for potential setbacks.

Following this exercise, groups presented their ideas in a plenary session (20 minutes, led by Sylvia Koukounidou), with each group’s representative summarising their concept in up to two minutes.

The workshop concluded with a wrap-up and a presentation on SCOPE and HumetricsHSS by Fotis Mystakopoulos (5 minutes), encapsulating the session’s insights and future directions for responsible research assessment.

5. Workshop results

A total of 48 people registered and about 25 people showed up. The size of the groups was uneven: the library group was the biggest, and the researcher and the policy maker groups contained only two people each. Maybe some more instructions on how to choose a group could be helpful to avoid this kind of situation. All groups filled in their sheets and presented the results of their discussions to the plenary. 

Regarding the results of the exercise are down to various interpretations, however, we are able to offer some key insights based on a lightweight analysis:

Chosen statements

A common thread runs through the selected statements, focusing on DORA and how to steer clear of the Impact Factor (IF) or identify ways to reduce reliance on metrics to measure quality and the impact of a research output. Some suggestions focus on assessing the methodology of research as part of formal assessment, which is strongly linked to the Hong Kong Principles philosophy, ensuring integrity and reproducibility of research. You find all the statements of the four initiatives in our Zenodo Publication.

Allies

For any success to be achieved in the implementation of statements identified by the participants, we always rely on allies, and we asked the participants to identify them: it is no surprise that researchers are mentioned throughout, given that the assessment reform requires them to endorse those changes. Research infrastructures are also mentioned as it is important to have supporting structures for the reform to be implemented. Other key allies are libraries, as they provide already a lot of support to researchers, and funders as they can influence how grants are being used. More specific allies were also identified, for example, policy makers which are key to addressing this issue from a policy perspective.

Risks

Finally, it is important to identify risks: a key cultural problem is the resistance to change. That is evident from various other efforts such as the implementation of Open Access practices. Other risks include  difficulty with alignment of efforts given how many parts of the system have to change simultaneously. Researcher mobility means that many different institutions have to work together towards common goals. Finally, another risk could be the lack of time, or the effort required to make these changes in research assessment reform. Researchers, while have been identified as allies in this process, must also balance work with doing research, teaching and other activities for their community.

These are insights provided by the SIG advocacy group, and is not a detailed analysis of the transcripts. We invite you to assess the statements on their merits and make your own conclusions.

6. Link to useful material

The most fundamental outcome of this workshop would be for others (participants and beyond) to be able to replicate this workshop to achieve further advocacy and change in their institutions. To that end, we create a Zenodo upload that includes the key documents anyone would need to prepare a similar workshop.

Zenodo Materials

  1. Presentation slide deck to reuse the structure and/or the contents. 
  2. Concept sheet – the sheet for the participants to provide a structured response for the exercises
  3. The spreadsheet that includes the statements of the four declarations/initiatives. It includes a breakdown of the full statements, the statements organised per stakeholder group as part of the preparation exercise, and the answers from the actual participants. It’s an interesting document to see the progression of how the statements are viewed by various professionals.

7. About the OPERAS Advocacy SIG

The OPERAS Advocacy SIG advocates for open scholarly communication in social sciences and humanities (SSH) and promotes the interests of SSH in policy and infrastructure development related to open scholarly communication.

The Advocacy SIG has produced several documents / white papers:

The OPERAS Advocacy SIG works on a yearly topic together with experts from the respective fields. The topic for the academic year 2024/2025 will be Diamond Open Access.

References: 

Curry, S., de Rijcke, S., Hatch, A., Pillay, D. (Gansen), van der Weijden, I., & Wilsdon, J. (2020, November 18). The changing role of funders in responsible research assessment: progress, obstacles and the way ahead. Rori.figshare.com. https://doi.org/10.6084/m9.figshare.13227914.v2 

de Rijcke , S., Cosentino , C., Crewe , R., D’Ippoliti, C., Motala-Timol , S., Binti A Rahman , N., Rovelli , L., Vaux, D., & Yupeng , Y. (2023). The Future of Research Evaluation: A Synthesis of Current Debates and Developments. https://doi.org/10.24948/2023.06

Helsinki Initiative on Multilingualism in Scholarly Communication (2019). Helsinki: Federation of Finnish Learned Societies, Committee for Public Information, Finnish Association for Scholarly Publishing, Universities Norway & European Network for Research Evaluation in the Social Sciences and the Humanities. https://doi.org/10.6084/m9.figshare.7887059.

Tregoning, J. (2018). How will you judge me if not by impact factor? Nature, 558, 345–345. https://doi.org/10.1038/d41586-018-05467-5