Journal Impact Factors - Association for Psychological Science

"Every summer my e-mail is enlivened by people and organizations writing about the latest journal impact factors (IF). Because I chair the APS Publications Committee, I have always done my best to feign deep interest about IFs. I know many people take them very seriously, but the truth is, I have never cared about them too much, although I do look at them. I certainly take citations seriously; they indicate at least to some degree the impact and worth of a paper. But journal impact factors seem to have been created by people who were a bit shaky on both arithmetic and descriptive statistics (the kind you learn in the first two weeks of baby stats classes in psychology). First, the arithmetic problem. IFs always go to three decimal places, to thousandths of a citation, giving a spurious impression of precision. The most recent IF for Psychological Science  (PS) is 4.543. Consider, however, its derivation: The number of citations in 2012 to articles published in the previous two years (a whole number, of course — it was 2344 for PS), divided by the number of articles published in PS in those two prior years (another whole number — 516). So dividing 2344 by 516 gives the IF. But by the rules of rounding we all learned somewhere around fourth grade — round to one decimal place more than the raw data — the proper IF is 4.5. Rounding to two places is sometimes done — 4.54 in this case — but why would anyone ever round to three decimal places — to thousandths — after dividing one whole number by another one? I have no answer. Why not, by this algorithm, use 4.5426356 as IF? After all, my calculator gives me this many digits to the right of the decimal point when I divide 2344 by 516. In many fields, journal IFs do not differ much among journals or from year to year, so using three decimal places also gives an impression of change where little or none exists. For example, the IF for Psychological Science 'skyrocketed' from 4.431 in 2011 to 4.543 in 2012 (or from 4.4 to 4.5 when rounded appropriately). Rounded to one decimal place, the humdrum constancy of the IF for most journals I have examined produces ennui. They just don’t change much. Yet when they do, even by a tiny bit, people get excited ... Carrying to three decimal places is the least of the problems with the IF. More critical is the fact that its creators apparently missed the first two weeks of Basic Statistics: If a distribution is strongly skewed, the mean of the distribution provides a measure of central tendency that can be greatly affected by some dramatic outliers. Of course, the IF represents the mean impact. I have been examining (and sometimes collecting) citations for one reason or another for 35 years, and their distributions are always staggeringly skewed in a positive direction. Most scientific papers are not cited much, but a few are cited a huge number of times. Plot the distribution and it will always be strongly skewed to the right (i.e., the bulk of the numbers fall on the left, often between 0 and 3 on a yearly basis). The tail of the distribution accounts for most of the variability in IF among journals. Thus, when one considers a change in the IF across years, this may be due to the presence (or absence) of just a few highly cited papers in the last two years, not an overall shift in the distribution. For Psychological Science in 2012, the median impact score is 3.0 and the mode is 0.0. (Yes, the most frequent number of citations for the articles published in 2010 and 2011 was zero — for about 15 percent of the papers.) The highest cited paper was the 'false positive' paper by Simmons, Nelson, and Simonsohn published in 2011 (69 citations). Perhaps it caused the uptick in IF for 2012; the next highest paper was from 2010 and was cited 27 times ... Because of the way IFs are calculated, total impact of journals is completely missed. Let’s consider the top two general psychology journals in terms of impact factor for 2012. These are almost always the same two or three titles, which are review/theory (versus empirical) journals. For 2012 they are Psychological Bulletin (15.575) andAnnual Review of Psychology (15.265), so by this measure the impact of these two is about the same. They finished in a dead heat (many years Annual Review is number one). But really? Let’s look at total citations in 2012 instead. The total citations for Psychological Bulletin were 30,814; those in Annual Review were cited 'only' 10,635 times (still quite good). Which outlet actually has the greater overall impact on the field? Why not look at the total citations of a journal? The Annual Review of Psychology is a great publication, but it is, in some ways, an odd one to include as a 'journal.' It is published once a year, the editorial team handpicks the review topics and the authors, and they often try to commission reviews of hot, breaking-news topics.I served on the editorial board for six years and saw the process