{"id":1869,"date":"2012-08-19T17:47:28","date_gmt":"2012-08-19T17:47:28","guid":{"rendered":"http:\/\/occamstypewriter.org\/scurry\/?p=1869"},"modified":"2012-09-08T13:56:14","modified_gmt":"2012-09-08T13:56:14","slug":"sick-of-impact-factors-coda","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/scurry\/2012\/08\/19\/sick-of-impact-factors-coda\/","title":{"rendered":"Sick of Impact Factors: Coda"},"content":{"rendered":"

My ‘Sick of Impact Factors<\/a>‘ blog post seems to have struck much more of a chord than I anticipated. At the time of writing it has attracted\u00a0over 12,900 page views and 460 tweets, far higher than my usual tallies.<\/p>\n

The post also generated over\u00a0130 comments, which is a daunting number for anyone who now stumbles across the post; but since the comment thread contains several useful ideas and insights from scientists and journal editors, I wanted to highlight them in a summary post.<\/p>\n

 <\/p>\n

Useful links<\/strong><\/p>\n

Several people flagged up papers and reports that I had not previously spotted. It is just this sort of productive interaction that makes the blog form so wonderful.<\/p>\n

Ross Mounce pointed out that he had previously discussed the non-utility of Journal Impact Factors (JIFs<\/a>) and and linked to a useful opinion piece by Per Seglen<\/a>.<\/p>\n

Tom Webb mentioned a paper by Brian D. Cameron that provides the\u00a0historical context for the JIF discussion<\/a>.<\/p>\n

Jason Priem, one of the authors of the Altmetrics manifesto pointed to the Total Impact web-site<\/a> (one attempt to put altmetrics into action)\u00a0and to a Chronicle of Higher Education article on alt metrics<\/a>.<\/p>\n

But for me, Stephan Schleim provided\u00a0the most interesting link, to a revealing comparison<\/a> of means and medians between Nature<\/em> and two Psychology journals. It provides a great example of the futility of JIFs in their current form and is a must-read#<\/sup><\/strong><\/span><\/em>.<\/p>\n

There were also interesting commentaries on the post from Telescoper<\/a>, Bjorn Brembs<\/a>, DrugMonkey<\/a> and Tom Webb<\/a>.<\/p>\n

 <\/p>\n

Concerns about metrics<\/strong><\/p>\n

<\/strong>Richard van Noorden wondered if it is the fixation on metrics<\/a> that is the root of the problem and suggests that article-level metrics are not likely solve the problem. This concern was shared by others (Stephen Moss was at particular pains<\/a> to emphasis the need to actually read papers when assessing them and their authors) and I would agree that the use of any metric runs this risk. However, the particular problem at the moment is the fixation on a single metric that is of no statistical merit.<\/p>\n

David Colquhoun cautioned that there is no evidence<\/a> that the use of metrics of any kind is a reliable way to judge scientific quality; however, as discussed in the comment thread (e.g. <\/em>by Chris Chambers<\/a>), there is no easy way to do a proper experiment on this.<\/p>\n

It came to light from comments on the post and on Twitter that some funding agencies and university faculties require<\/em> JIFs to be included in lists of publications applications for funding or promotion. One commenter<\/a> was explicitly instructed by his HoD to include JIFs to make it easier for the person assessing his application. There was no keener illustration of the idiocy of the processes to which the research community has to endure. I find myself wondering what the general public would make of this.<\/p>\n

Some argued that JIF should at least be maintained as a fair measure of journal ranking but this view ignores the gaming and trickery that goes on (see comments by David Howey<\/a>\u00a0and\u00a0Bob O’Hara<\/a>), the inappropriateness of a mean for such a skewed distribution and the arbitrariness of the two-year time window which seriously undervalues some journals\/fields (see especially the\u00a0link<\/a> posted by Stephan Schleim).<\/p>\n

Steve Caplan suggested<\/a> that people will always want to aim for ‘high-ranking’ journals, whether or not they are judged by impact factor. I sense this view is widely held but would like to suggest that the rise of PLOS ONE shows that the landscape is changing. Rather than rely on a journal brand (and suffering the delays of working our way down the rankings during manuscript submission), we should be aiming to get good quality work published quickly and after peer review that only focuses on the quality and coherence of the manuscript, rather than on guesses about likely future significance. The community is well-placed to judge significance after publication; the key \u2014 and non-trivial \u2014 question is how to go about that.<\/p>\n

 <\/p>\n

What alternative metrics (altmetrics)\u00a0might\u00a0we use instead of the JIF?<\/strong><\/p>\n

<\/strong>Setting aside for a moment the concerns listed above, no-one (myself included) seemed to have clear ideas on what metrics to use, though there is plenty of development work going on in this area. Some wondered whether tweets or Facebook likes should be counted as a meaningful statistic but there is little enthusiasm for such measures, in large part because they are effortless to make and so run the risk of representing snap or ill-considered judgements.<\/p>\n

I think the key will be to gather information from people that we trust<\/a>. That trust needs to be established over time (as on Twitter) by a consistent pattern of linking to quality material or making thoughtful comments. Development of unique identifiers (as suggested by Cromercrox<\/a>) might enable us to build a community like the Faculty of 1000<\/a> but much bigger and offering greater coverage.<\/p>\n

But the whole business of metrics remains rather foggy.<\/p>\n

 <\/p>\n

What action should we take?<\/strong><\/p>\n

It was great to have such a vigorous response to my post but I would like all of us who have taken an interest in this issue to make sure that we take action of some sort. Can I suggest that we:<\/p>\n