Bias in Open Science Advocacy: The Case of Article Badges for Data Sharing | Absolutely Maybe

"The source of this was a paper by Mallory Kidwell and colleagues in PLOS Biology, called 'Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency'. The developer and prominent social marketer of the badges is the senior author of that paper and head of the organization tied to them, Brian Nosek.

The title has most of the authors’ strong claims embedded in it. They are:

1. The intervention studied is simply offering a badge if authors meet certain criteria.
2. The intervention is 'low cost', 'low risk', 'relatively resource-lite'.
3. There are no other repercussions – 'if badges are not valued by authors, they are ignored and business continues as usual'.
4. Badges are 'dramatically' effective – dramatic(ally) appears 9 times in the paper.
5. Badges increase authors’ use of the kind of repository the badges exist in part to promote – like that of Nosek & co’s Open Science Framework (OSF).

The paper reports what seems to be the only study on this. It involves looking back at what happened before and after a prominent journal, Psychological Science, introduced badges in January 2014. To gauge whether any changes observed might be happening in the discipline anyway, the authors chose 4 journals that did not have badges for comparison.

They then collected data on open-ness of data and study materials from all 5 journals for research-based articles from 2012 to May 2015: 2 years of 'before' data and 17 months of 'after' data. Let’s look at the authors’ 5 claims and see whether the data support them. The open data for the project were a big help, so the authors get lots of brownie points for walking the open data talk."