The Problems with Shen/Björk’s “420,000” « Walt at Random

"Cenyu Shen and Bo-Christer Björk published '‘Predatory’ open access: a longitudinal study of article volumes and market characteristics' in BMC Medicine 13, October 2015. (I’m bemused at the idea that this is a medical paper, but that’s a separate discussion.) I started questioning the paper’s conclusions as soon as it appeared, and continued to do so in my blog and in Cites & Insights.

Quite apart from the apparent assumption that Beall’s word is gospel when it comes to journals being 'predatory'—an assumption I found, and find, appalling—I thought the numbers were implausible. The authors used a sample of 613 journals to assert that there were around 8,000 active 'predatory' journals in 2014 and that those journals published around 420,000 articles in 2014 (up from around 310,000 in 2013 and 212,000 in 2012).

Being presented with a case for the implausibility of the numbers, the authors responded that the article was peer-reviewed and used proper statistical methods. As I was writing this, I took the time to read open reviewer comments on the article and the authors’ responses. Notably, all of the reviewers said they weren’t qualified to review the statistics—and there were certainly questions raised about the assumption that to be on Beall’s list was to be predatory.

The authors are right about one thing: looking at all the journals is a ridiculously large task. But that task showed that gray journals are just as heterogeneous as I thought they were, making it easy for a 6% sample to be wildly off base.

[...]

What Went Wrong?

How could these two scholars be so far off? First there’s the assertion that all journals on Beall’s lists are actually predatory. Second, the 'stratified' random sampling method involves some tricky assumptions, based on a 'suspicion' that was 'verified' by sampling all of ten journals—the suspicion 'that journals from small publishers often publish a much higher number of articles than those of large publishers.'

The sampling used in this study yielded a much lower percentage of empty journals than my 100% survey. The article estimates that 67% of listings represent active journals; my 100% survey (admittedly of a larger list) shows 40% active journals. That’s an enormous difference: instead of 8,000 active journals from the smaller list, you wind up with around 4,800. That’s probably about right (I show 5,988—but that’s from a much larger list).

Beyond that, it appears that the sheer heterogeneity of journals makes projection from a small sample so dicey as to be useless. Unfortunately, I believe that to be the case."