D-Lib Magazine In Brief and In the News: NISO Altmetrics Project: Standardizing New Alternative Assessment Metrics

"As long as there have been authors, there have been people interested in quantifying the impact or performance of the work those authors have created. There are a variety of metrics that were used as proxies of impact long before our modern scholarly communication's ecosystem developed. These measures included things like individual sales, a publication's circulation, the reaction of luminaries, and the support of institutions through researcher promotion and tenure decisions. Scholarly impact began focusing intently on bibliometrics and citation study in the late 1950s with Eugene Garfield's work and the subsequent launch of the Journal Impact Factor (IF), now owned by Thomson Reuters. The challenges with the Impact Factor are well documented and include self-citation, time delay in producing data, and the fact that it is a publication-level metric, not a metric of the author or the article. As content distribution for scholarship has moved toward online distribution and a greater focus on the individual article, a variety of new metrics have developed and become commonplace in our community. Download counts, through COUNTER statistics, became one important metric, but again, these are primarily journal metrics, not individual item metrics. Additionally, download counting is subject to the same potential challenges as the IF, such as gaming and a focus on popularity over substance. Most traditional metrics focus on journals and their content, but they leave out a wide variety of non-traditional content such as datasets, software, visualization tools, or performance recordings, which are as, or even more, important than the journal article in some disciplines. Many new forms of scholarly communication are gaining traction and researchers frequently turn to new forums for scholarship discussion. Blogs, Facebook, Twitter, Mendeley, Figshare, various repositories, and many other start-ups are rapidly becoming places where researchers share their work and discuss the work of others in their fields. These forums are generating a great deal of ancillary data about use and impact. However, as many service providers have grown to serve this assessment need, there is a great deal of inconsistency about the data gathered and how it compares across these different services. Scott Chamberlain, in his Information Standards Quarterly article, 'Consuming Article-Level Metrics: Observations and Lessons', identified the challenges of comparing metrics from a variety of service providers, since their source data isn't always equivalent. Into this environment, the National Information Standards Organization (NISO) has launched an initiative to determine the need for, priorities of, scope, and definition of standards and best practices related to new forms of scholarly assessment, often called Altmetrics. With the generous funding of the Alfred P. Sloan Foundation, NISO began a two-phase initiative in 2013. The first phase is a series of in-person and virtual conversations to brainstorm and prioritize altmetrics issues that could be addressed with standards or best practices. In the second phase, working groups will be established to develop the identified standards or recommended practices. The first of three in-person meetings was held in San Francisco, CA in October 2013; a second was held in Washington, DC in December 2013; and a third is planned for Philadelphia on January 23, 2014, prior to the American Library Association Midwinter meeting. The meetings follow a common agenda of lightning talks on existing projects, brainstorming activities, and small group break-outs to discuss the proposed ideas. In the first meeting, break-outs focused on definitions, quality and data science, and business and use cases. In the second meeting, groups addressed definitions, stakeholder values, existing research on assessment, quality versus quantity, granularity of measurement, assessment and discovery, tools and community engagement, future proofing solutions, and identity management. Written notes, summaries of the breakout discussions, and video recordings of each meeting are available from the project webpage at: http://www.niso.org/topics/tl/altmetrics_initiative/ ..."