{"id":2436,"date":"2013-10-17T10:10:07","date_gmt":"2013-10-17T09:10:07","guid":{"rendered":"http:\/\/occamstypewriter.org\/scurry\/?p=2436"},"modified":"2013-10-22T15:35:56","modified_gmt":"2013-10-22T14:35:56","slug":"impact-factors-are-clouding-our-judgement","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/scurry\/2013\/10\/17\/impact-factors-are-clouding-our-judgement\/","title":{"rendered":"Impact factors are clouding our judgement"},"content":{"rendered":"

Nature <\/em>has\u00a0an interesting news feature this week on impact factors. Eugenie Samuel Reich’s article<\/a> \u2014 part of a special supplement<\/a> covering various aspects of the rather ill-defined notion of\u00a0impact\u00a0\u2014 <\/em>explores whether publication in journals such as Nature<\/em> or Science<\/em> is a game-changer for scientific careers.<\/p>\n

The widely-held assumption is that they are. And the stories from young scientists interviewed by Reich*, who had almost all published in Nature<\/em> or Science<\/em> or Cell<\/em> back in 2010, would appear to confirm that. Their papers in prestige journals won jobs or grants or opened doors to clinical trials that has previously been shut. Or at least that’s what they assume or believe or feel; no-one can quite be sure because the rules of the game are unspoken and unwritten.<\/p>\n

\"Nature<\/a><\/p>\n

 <\/p>\n

But the trouble is that these unofficial rules appear nigh on universal. I was certainly mindful of them when I embarked on my research career more than twenty years ago (as I mentioned in my contribution to this week’s\u00a0Nature Podcast<\/a>). Regular visitors to this blog will be aware that since then I have modified my views and now see the excessive influence of impact factors as a kind of addiction<\/a> for which the scientific community needs to find a cure.<\/p>\n

It can be a hard argument to pitch because the culture of dependence is so embedded. The lure of high-impact journals is strong and underpinned by some rational motivations. As noted by\u00a0Finnish scientist\u00a0Annele Virtanen, one of \u00a0Reich’s interviewees, the competition for publication in Nature<\/em> of Science<\/em>\u00a0acts as a spur for scientists to be ambitious in their research. No bad thing of course but the trouble sets in when rewards are tied too closely to the particular achievement of a Nature<\/em> or Science<\/em> paper.<\/p>\n

While it is certainly true that many of the papers published in these journals are of very high quality \u2014 and that on average they garner more citations than papers in journals focused on particular disciplines \u2014 it is too often forgotten that the impact factor not only disguises the very real variation in performance of papers in any one title but flatters them because its dubious method of calculation skews the measure of average performance significantly towards the higher end of the distribution.<\/p>\n

The scientific community too often overlooks the granularity of the data and thinks only in terms of impact factors. Thus everyone who gains entry to Nature<\/em> or Science<\/em> wins an impact factor prize \u2014 irrespective of the actual significance of their work; and of course, those who narrowly fail to make the grade (in an assessment process that is highly stochastic) lose out. Virtanen, the recent beneficiary of this system, says “I can’t see so many bad sides” \u2014 a common enough view. But would she prefer to trust her future prospects to the uncertainty of getting her next paper through the narrow doors of the very top journals or would it be better to be able to rely on a system that does a more rigorous and fairer job of assessing what she has actually done?<\/p>\n

There are positive moves in this direction. Sandra Schmid, head of cell biology at the University of Texas South-western Medical School, has recently taken steps to move away from the over-reliance on impact factors<\/a> in her hiring procedures. I am pushing for similar revisions at my own institution. These moves chime with the recent San Francisco Declaration on Research Assessment<\/a> which hopes to encourage all stakeholders in the scientific process \u2014 universities, funders, publishers and learned societies \u2014 to revise and improve their methods of assessment, specifically to eliminate the unhealthy lure of the impact factor. In the UK, the Wellcome Trust has long had a stated policy of not considering where<\/em> applicants’ work is published when judging grant proposals, a policy now adopted by the UK Research Councils<\/a>. But it is one thing to formulate a policy; quite another to make it work.<\/p>\n

In the editorial<\/a> accompanying this week’s supplement on impact Nature<\/em> restates its long-standing opposition to the mis-use of impact factors in judging individuals or individual papers (a position that has been usefully repeated in other Nature<\/em>-branded titles such as Nature Chemistry<\/a><\/em> and Nature Materials<\/em><\/a>). This is laudable but insufficient. It sits uneasily with the full-page adverts that appear annually with the announcement of yet another incremental rise in impact factor. This recent one<\/a>, trumpeting Nature’s 2011 IF of 36 is accompanied by the strap-line ‘Energize your scientific career\u2026’. What are researchers to make of that if not a repetition of the unspoken mantra that publication in top-tier journals is vital to real success in science?<\/p>\n

The editorial also repeats the line that the obsession with impact factors is a problem for the scientific community to address. And that is true \u2014 it is a largely self-inflicted problem and it is primarily our<\/em>\u00a0responsibility to sort it out. But I find it odd that Nature<\/em> appears to see itself as apart from that community, especially when its editors and reviewers are drawn from within it. I don’t think the dividing line is so easy to discern \u2014 especially given the supportive commentary of some Nature<\/em> journal editors.<\/p>\n

Where I do agree strongly with the editorial line is in the declared need for “for research evaluators to be explicit about the methods they use to measure impact.” In this, Nature<\/em>, and indeed all scholarly journals can help \u2014 and at negligible cost. I call for them to publish all the data on which their impact factor calculations are based. Every year, when the new impact factor is released and advertised, please also publish the citation numbers and distributions on which it is based<\/strong>. This transparency will help to demystify the magical lure of that one number by revealing a truer picture of the performance of all the papers that contribute to it. It will reveal the variation in granular detail \u2014 the big hitters and<\/em> the damp squibs. Comparison between journals would be enriched because the real overlap in the citation distributions \u2014 too often forgotten in the obsession with just one number \u2014 would be made evident.<\/p>\n

PLOS is already leading the way<\/a> in making this type of information available. Nature<\/em> could do a tangible and valuable service to the scientific community by a simple act of transparency. It could blow away some of the clouds that are presently obscuring our judgement.<\/p>\n

Now that I come to think of this proposal, I can’t see any bad sides.<\/p>\n

Update (20-10-2013, 15:50) \u2014<\/span>\u00a0<\/strong>If I’d had the time I would have read all of the articles in Nature<\/em>‘s supplement before publishing this post and made a point of including a link to the piece by David Shotton<\/a> on efforts to make citation data open.<\/p>\n

*Update (22-10-2013, 15:33) \u2014<\/span><\/strong> The original version of this post referred to the Nature author as Eugenie Samuel Reich, which is her full name; but she kindly informs me that her surname is simply Reich. The text had been modified to reflect this. Apologies for any confusion.<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Nature has\u00a0an interesting news feature this week on impact factors. Eugenie Samuel Reich’s article \u2014 part of a special supplement covering various aspects of the rather ill-defined notion of\u00a0impact\u00a0\u2014 explores whether publication in journals such as Nature or Science is … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[152,107],"tags":[208,145],"class_list":["post-2436","post","type-post","status-publish","format-standard","hentry","category-open-access-2","category-scientific-life","tag-impact-factor","tag-open-access"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2436","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/comments?post=2436"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2436\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/media?parent=2436"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/categories?post=2436"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/tags?post=2436"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}