Beyond Citation-Counting: Metrics and Altmetrics for Demonstrating Scholarly Impact

"I would be remiss if I didn’t talk about the limitations of metrics, because we should understand both the strengths and the shortcomings when we engage with these kinds of assessment tools.

First of all, it pays to maintain perspective on any metrics based in citations, because it isn’t always clear why authors cite and what the inclusion—or exclusion—of a citation is really meant to communicate....

But beyond just citation-based metrics, we should also acknowledge the limitations of research metrics more broadly....

They are all inherently backward-looking. Or if you’ve ever heard a radio advertisement for an investment opportunity or mutual fund, you’ve probably heard that fine print at the end: “Past performance is not an indicator of future results.” ...

[Manipulation:] Potential manipulation of a metric, or “gaming the system” usually just requires motivation. Citation-based metrics could be manipulated by means such as self-citation, reciprocal citation deals between authors, or similar. Altmetrics such as views or downloads could be manipulated through bots, incentivizing others to boost the appearance of attention, a professor requiring students to download his work, or other means. 

[Context/interpretation:] Furthermore, any number can be used out of context, misconstrued, or misinterpreted. 

[Absence of metrics:] Sometimes there are simply gaps in what numbers can be obtained, but the absence of readily available metrics does not automatically equate to the absence of impact, a fact of which I frequently try to reassure individual researchers. 

[Journal prestige vs. economics of access:] When researchers feel pressured to publish only in high-impact journals, which tend to be very expensive, who is being excluded? Researchers at small institutions with smaller library budgets, researchers in developing countries, and others will not have access to the work in these scholarly journals, so they are excluded from the conversation we create. 

[Philosophy:] Fundamentally, there is also an argument that the over-emphasis on arbitrary measurements to justify research as “worthwhile” undermines the philosophy that the pursuit of knowledge has intrinsic value. If a work is not cited within 2 years – or within 20 years – does that indicate that the work has no value? That it does not have the capacity for future value? That the researcher is not still worthy of recognition for their pursuits and accomplishments? ..."