The challenge of measuring scholarly communication | Research Information

"The problem is, of course, not that there are things that metrics just can’t tell us – but that there are a lot of unrealistic expectations around what metrics should be able to show and how quickly solutions can be found. 

It’s not just that governments want to find ways of improving the socio-economic return on investment from taxpayer funded research, but part of a whole shift in the use of metrics in society, with league tables promising to show how an ever increasing number of different types of organisations compare. In such an environment the notion that there may be limits to what can be meaningfully quantified, or the speed with which solutions can be found, is incomprehensible. ...

The broadening from wanting to understand the academic impact of research to also wanting to understand the socio-economic impact of research is to be welcomed, but while we undoubtedly have access to more data than ever before it’s not clear that we are any closer to getting a true picture of scholarly communications. 

The temptation to overly reduce the complexity of the research ecosystem will always be there for the purposes of policy making, management decisions, or the increasingly ubiquitous ranking tables, but there are no quick fixes to the problem of measuring research impact. 

Meaningful metrics have to reflect the complicated and diverse nature of the research, and they will only emerge when all the different parties work together to discover what is important to their particular community. 

We are a long way from getting simple answers, but we are increasingly able to see the complexity of the problem."