Modelling the Effects of Open Access, Gender and Collaboration on Citation Outcomes: Replicating, Expanding and Drilling

"Previous studies have demonstrated that open access (OA), international collaboration and the gender composition of research teams are each individually connected to citation outcomes; however, these dimensions are also inter-related. Only one small-scale study to date has examined the question of whether each of these three parameters continues to have an influence on citation when controlling for the others (Struck, Roberge, & Campbell, 2017). The present paper has three objectives.

1. Assess the robustness of the results of the previous, small-scale study by attempting to replicate the results. The replication will be conceptual rather than exact (Velden, 2017): a different data source is used here than in the original study, the indicators are parametrized in a different way, and several modelling approaches are compared.

2. Expand this analysis approach to a much wider scope. Whereas the previous study examined the dynamics of only two subfields of research, the present work assesses dynamics across all subfields (except the Arts & Humanities, see footnote 7). Furthermore, this study will cover the 2008–2012 period and all papers worldwide (rather than just papers from 2012 involving a US author).

3. Finally, “drill down” into subsamples of the overall data set to extract policy-relevant contextual information, comparing a citation benefit in a local context to the benefit “in general.” This type of contextualization can move policy discussions from discussions of global trends to whether a particular intervention would be beneficial “to us” within a local context....

The previous study focused on two subfields: Cardiovascular System & Hematology, and Developmental Biology. That analysis drew the following conclusions:

a) OA had a significant and meaningful effect on citation scores, in both subfields.

b) International collaboration had a significant and a meaningful effect (albeit smaller than that for OA) in Cardiovascular System & Hematology. In Developmental Biology the effect was not statistically significant.

c) Gender composition had no significant effect in Cardiovascular System & Hematology. In Developmental Biology, the effect was statistically significant, with citation scores highest for mixed-gender teams. Because of the way in which the indicator was parametrized, identifying the optimal gender balance (optimal from a citation perspective) was not possible in a thorough fashion, though mixed-leaningmale seemed to have the highest citation effect. The effect size was much smaller for gender composition than it was for OA.

The replication presented here reached broadly similar conclusions:

a) OA had large and statistically significant effects in both subfields. In Cardiovascular System & Hematology, the effect of gold OA was particularly pronounced. These findings held across all three models.

b) International collaboration had a significant effect in both subfields, though it was smaller in Developmental Biology, where previous results were not statistically significant. Furthermore, the effect of international collaboration was once again smaller than that of OA. These findings held across all three models.

c) Gender composition showed statistically significant results across all three models in Developmental Biology: the more women involved in the research team, the lower the citation score was predicted to be, with decreases of up to 30%. This finding contradicted previous results, which suggested (based on a very coarse approach to parametrization) that mixed-gender teams were optimal. In Cardiovascular System & Hematology, only the robust regression model showed statistically significant results: an increase in citations with an increase in the proportion of women in the research team....

The wider analysis showed that the OA citation advantage holds generally, with green OA offering a much larger benefit than gold OA. The larger the proportion of women in research is, the lower the predicted citation outcomes. International collaboration offers a citation advantage, smaller than the advantage of either green or gold OA.... "