bjoern.brembs.blog » Even without retractions, ‘top’ journals publish the least reliable science

"Vox health reporter Julia Belluz has recently covered the reliability of peer-review. In her follow-up piece, she asked 'Do prestigious science journals attract bad science?'. However, she only covered the data on retractions, not the much less confounded data on the remaining, non-retracted literature. It is indeed interesting how everyone seems to be attracted to the retraction data like a moth to the flame. Perhaps it’s because retractions constitute a form of ‘capital punishment’, they seem to reek of misconduct or outright fraud, which is probably why everybody becomes so attracted – and not just journalists, scientists as well, I must say. In an email, she explained that for a lay audience, retractions are of course much easier to grasp than complicated, often statistical concepts and data. However, retractions suffer from two major flaws which make them rather useless as evidence base for any policy: I. They only concern about .05% of the literature (perhaps an infinitesimal fraction more for the ‘top’ journals II. The data are confounded by error-detection variables that are hard to trace. Personally, I tentatively interpret what scant data we have on retractions as suggestive that increased scrutiny may only play a minor role in a combination of several factors leading to more retractions in higher ranking journals, but I may be wrong. Indeed, we emphasize in several places in our article on precisely this topic that retractions are rare and hence one shouldn’t place so much emphasis on them ..."