{"id":2811,"date":"2015-06-23T08:44:16","date_gmt":"2015-06-23T07:44:16","guid":{"rendered":"http:\/\/occamstypewriter.org\/scurry\/?p=2811"},"modified":"2015-07-08T18:38:54","modified_gmt":"2015-07-08T17:38:54","slug":"data-not-shown-time-to-distribute-some-common-sense-about-impact-factors","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/scurry\/2015\/06\/23\/data-not-shown-time-to-distribute-some-common-sense-about-impact-factors\/","title":{"rendered":"Data not shown: time to distribute some common sense about impact factors"},"content":{"rendered":"

It\u2019s that time of year when all clear-thinking people die a little inside: the latest set of journal impact factors has just been released<\/a>.<\/p>\n

Although there was an initial flurry of activity on Twitter last week when the 2015 Journal Citation Reports* were published by Thomson Reuters, it had died down by the weekend. You might be forgiven for thinking that the short-lived burst of interest means that the obsession with this damaging metric is on the wane. But this is just the calm before the storm. Soon enough there will be wave upon wave of adverts and emails from journals trumpeting their brand new impact factors all the way to the ridiculous third decimal place. So now is the time to act \u2013 and there is something very simple that we can all can do.<\/p>\n

For journals, promotion of the impact factor makes a kind of sense since the number \u2013 a statistically dubious calculation of the mean number of citations that their papers have accumulated in the previous two years \u2013 provides an indicator of the average performance of the journal. It\u2019s just good business: higher impact factors attract authors and readers.<\/p>\n

But the invidious effects of the impact factor on the business of science are well-known and widely acknowledged. Its problems have been recounted in detail on this blog<\/a> and elsewhere<\/a>. I can particularly recommend Steve Royle’s recent dissection<\/a> of the statistical deficiencies of this mis-measure of research.<\/p>\n

There is no shortage of critiques but the impact factor has burrowed deep into the soul of science and is proving hard to shift. That was a recurrent theme of the recent Royal Society meeting on the Future of Scholarly Scientific Communication<\/a> which, over four days, repeatedly\u00a0circled back to the mis-application of impact factors as the perverse incentive that is at the root of problems with the evaluation of science and scientists, with reproducibility, with scientific fraud, and with the speed and cost of publishing research results. I touched on some of these issues in a recent blogpost<\/a> about the meeting; (you can listen to recordings of the sessions<\/a> or read a summary<\/a>).<\/p>\n

The Royal Society meeting might have considered the impact factor problem from all angles but \u00a0discovered once again \u2013 unfortunately \u2013 that there are no revolutionary solutions to be had.<\/p>\n

The San Francisco Declaration on Research Assessment (DORA)<\/a> and the Leiden Manifesto<\/a> are commendable steps in the right direction. Both are critical of the mis-use of impact factors and foster the adoption of alternative processes for assessment. But they are just steps.<\/p>\n

That being said, steps are important. Especially so if the journey seems arduous.<\/p>\n

Another important step was made shortly after the Royal Society meeting by the EMBO Journal<\/em> and is one that gives us all an opportunity to act. Bernd Pulverer, chief editor of EMBO J<\/em>., announced<\/a> that the journal will from now on publish its annual citation distributions, which comprise the data\u00a0on which the impact factor is based. This may appear to be merely a technical development but it marks an important move towards transparency that should help to dethrone the impact factor.<\/p>\n

 <\/p>\n

\"EMBO<\/a><\/p>\n

 <\/p>\n

The citation distribution for EMBO J.\u00a0<\/em>is highly skewed. It is dominated by a small number of papers that attract lots of citations and a large number that garner very few. The\u00a0<\/em>journal publishes many papers that attract only 0, 1 or 2 citations in a year and a few that have more than 40. This is not unusual \u2013 almost all journals will have similarly skewed distributions<\/a>\u00a0\u2013 but\u00a0what it\u00a0makes clear are the huge variations in citations that the papers in any given journal attract. And yet all will be \u2018credited\u2019 with the impact factor of the journal \u2013 around 10 in the case of EMBO J<\/em>.<\/p>\n

By publishing these distributions, the EMBO Journal<\/em>\u00a0is being commendably transparent about citations to its papers. It\u00a0is not just a useful reminder that behind the simplicity of reducing journal performance to a single number is an enormous spread in the citations attracted by individual pieces of work.\u00a0As Steve Royle\u2019s excellent analysis reveals, the IF is a poor discriminator between journals and a dreadful one for papers. Publishing citation distributions therefore directs the attention of anyone who cares about doing evaluation properly back where it belongs: to the work itself. The practice ties in nicely with articles 4 and 5 of the Leiden Manifesto<\/a>.<\/p>\n

So what can you<\/em> do? Simple: if in the next few weeks and months you come across an advert or email bragging about this or that journal\u2019s impact factor, please contact them to ask why they are not showing the data on which the impact factor is based. Ask them why they are not following the example set by the EMBO Journal<\/em>. Ask them why they think it is appropriate to reduce their journal to a single number, when they could be transparent about the full range of citations that their papers attract. Ask them why they are not showing the data that they rightly insist authors provide to back up the scientific claims in the papers they publish. Ask them why they won\u2019t show the broader picture of journal performance. Ask them to help address the problem of perverse incentives in scientific publishing.<\/p>\n

*The title is somewhat confusing since the 2015 JCR contains the impact factors calculated for 2014.<\/p>\n","protected":false},"excerpt":{"rendered":"

It\u2019s that time of year when all clear-thinking people die a little inside: the latest set of journal impact factors has just been released. Although there was an initial flurry of activity on Twitter last week when the 2015 Journal … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[152],"tags":[],"class_list":["post-2811","post","type-post","status-publish","format-standard","hentry","category-open-access-2"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2811","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/comments?post=2811"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2811\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/media?parent=2811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/categories?post=2811"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/tags?post=2811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}