Growing Impact of Non-Elite Journals | The Scholarly Kitchen

"... When a paper like this comes to light, there is a always a temptation to begin debating its broader context before we can vet its methods. Like the paper claiming that scientists are reading fewer papers, journalists rarely put much effort into validating even the most basic numbers underlying the authors’ bold claims. It is much more newsworthy to discuss the paper’s significance. Instead of ruminating about the broader economic, sociological and technological changes that may underly Google’s findings, I’m going to focus on on their methods because I believe that their findings may simply be an artifact of the researchers’ strict definition of what constitutes an 'elite' journal. The Google team defined an 'elite' journal as one of the top 10 most-cited journals (based on their h-5 index) for each of their 261 subject categories and for each year between 1995 and 2013. Google researchers then identified the 1000 most-cited articles in each category-year and looked to see how many were published in the elite versus non-elite journals. Google’s method is simple and intuitive, but it contains two fundamental assumptions that they don’t bother to verify in their paper. First, the use of a top-ten list assumes that the production of scientific literature is constant from year to year. It isn’t. If we assume that the number of articles grows approximately 3% per annum, and we want to conserve the ratio between elite and non-elite journals, then by 2013, there should be 17 journals in their 'elite' group not 10. Second, the Google researchers also assume that elite journals publish roughly the same number of articles each year, which also doesn’t seem to be valid, at least for medicine ..."