My ‘Sick of Impact Factors‘ blog post seems to have struck much more of a chord than I anticipated. At the time of writing it has attracted over 12,900 page views and 460 tweets, far higher than my usual tallies.
The post also generated over 130 comments, which is a daunting number for anyone who now stumbles across the post; but since the comment thread contains several useful ideas and insights from scientists and journal editors, I wanted to highlight them in a summary post.
Several people flagged up papers and reports that I had not previously spotted. It is just this sort of productive interaction that makes the blog form so wonderful.
Tom Webb mentioned a paper by Brian D. Cameron that provides the historical context for the JIF discussion.
Jason Priem, one of the authors of the Altmetrics manifesto pointed to the Total Impact web-site (one attempt to put altmetrics into action) and to a Chronicle of Higher Education article on alt metrics.
But for me, Stephan Schleim provided the most interesting link, to a revealing comparison of means and medians between Nature and two Psychology journals. It provides a great example of the futility of JIFs in their current form and is a must-read#.
Concerns about metrics
Richard van Noorden wondered if it is the fixation on metrics that is the root of the problem and suggests that article-level metrics are not likely solve the problem. This concern was shared by others (Stephen Moss was at particular pains to emphasis the need to actually read papers when assessing them and their authors) and I would agree that the use of any metric runs this risk. However, the particular problem at the moment is the fixation on a single metric that is of no statistical merit.
David Colquhoun cautioned that there is no evidence that the use of metrics of any kind is a reliable way to judge scientific quality; however, as discussed in the comment thread (e.g. by Chris Chambers), there is no easy way to do a proper experiment on this.
It came to light from comments on the post and on Twitter that some funding agencies and university faculties require JIFs to be included in lists of publications applications for funding or promotion. One commenter was explicitly instructed by his HoD to include JIFs to make it easier for the person assessing his application. There was no keener illustration of the idiocy of the processes to which the research community has to endure. I find myself wondering what the general public would make of this.
Some argued that JIF should at least be maintained as a fair measure of journal ranking but this view ignores the gaming and trickery that goes on (see comments by David Howey and Bob O’Hara), the inappropriateness of a mean for such a skewed distribution and the arbitrariness of the two-year time window which seriously undervalues some journals/fields (see especially the link posted by Stephan Schleim).
Steve Caplan suggested that people will always want to aim for ‘high-ranking’ journals, whether or not they are judged by impact factor. I sense this view is widely held but would like to suggest that the rise of PLOS ONE shows that the landscape is changing. Rather than rely on a journal brand (and suffering the delays of working our way down the rankings during manuscript submission), we should be aiming to get good quality work published quickly and after peer review that only focuses on the quality and coherence of the manuscript, rather than on guesses about likely future significance. The community is well-placed to judge significance after publication; the key — and non-trivial — question is how to go about that.
What alternative metrics (altmetrics) might we use instead of the JIF?
Setting aside for a moment the concerns listed above, no-one (myself included) seemed to have clear ideas on what metrics to use, though there is plenty of development work going on in this area. Some wondered whether tweets or Facebook likes should be counted as a meaningful statistic but there is little enthusiasm for such measures, in large part because they are effortless to make and so run the risk of representing snap or ill-considered judgements.
I think the key will be to gather information from people that we trust. That trust needs to be established over time (as on Twitter) by a consistent pattern of linking to quality material or making thoughtful comments. Development of unique identifiers (as suggested by Cromercrox) might enable us to build a community like the Faculty of 1000 but much bigger and offering greater coverage.
But the whole business of metrics remains rather foggy.
What action should we take?
It was great to have such a vigorous response to my post but I would like all of us who have taken an interest in this issue to make sure that we take action of some sort. Can I suggest that we:
- Keep banging on about the futility and idiocy of using JIFs
- Get funders and university administrators to publicly disavow their use and issue clear guidance to panel members and reviewers. The Wellcome Trust has made a start (see last bullet point here).
- Encourage wider use of open peer-review, where reviewers’ and authors’ comments are published alongside papers; this may foster more online commentary post-publication*.
- Support the adoption of unique identifiers for authors, to be used when publishing or making any online comment on work — this should help to build reputations and trust among the commenting community*.
If anyone has additional suggestions, please add them to the comments.
#Update (20-8-2012; 21:10): But see comment from Pep Pàmies and ensuing discussion.
*These suggestions prompted by a comment from Cromercrox