{"id":2284,"date":"2013-05-16T19:01:19","date_gmt":"2013-05-16T18:01:19","guid":{"rendered":"http:\/\/occamstypewriter.org\/scurry\/?p=2284"},"modified":"2018-01-19T22:13:40","modified_gmt":"2018-01-19T21:13:40","slug":"impact-factors-declared-unfit-for-duty","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/scurry\/2013\/05\/16\/impact-factors-declared-unfit-for-duty\/","title":{"rendered":"Impact factors declared unfit for duty"},"content":{"rendered":"

Regulars at this blog will be familiar with the dim view<\/a> that I have of impact factors, in particular their mis-appropriation for the evaluation of individual researchers and their work. I have argued for their elimination<\/a>, in part because they act as a brake on the roll-out of open access publishing but mostly because of the corrosive effect they have on science and scientists.<\/p>\n

I came across a particularly dispiriting example of this recently when I was asked by a well-known university in North America to help assess the promotion application of one of their junior faculty. This was someone whose work I knew \u2014 and thought well of \u2014 so I was happy to agree. However, when the paperwork arrived I was disappointed to read the following statement the description of their evaluation procedures:<\/p>\n

“Some faculty prefer to publish less frequently and publish in higher impact journals. For this reason, the Adjudicating Committee will consider the quality of the journals in which the Candidate has published and give greater weight to papers published in first rate journals.”<\/p><\/blockquote>\n

Which means of course that they put significant weight on impact factors when assessing their staff. Given the position I had developed in public (and at some length) I felt that this would make it difficult for me to participate. I wrote to the institution to express my reservations:<\/p>\n

“\u2026I think basing a judgement on the name or impact factor of the journal rather that the work that the scientist in question has reported is profoundly misguided. I am therefore not willing to participate in an assessment mechanism that perpetuates the corrosive effects of assessing individuals by considering what journals they have published in. I would like to be able to provide support for Dr X’s application but feel I can only do so if I can have the assurance of your head of department that the Committee will work under amended criteria and seek to evaluate the applicant’s science, rather than placing undue weight on where he has published.”<\/p><\/blockquote>\n

The reply was curt \u2014 they respected my decision for declining. And that was it.<\/p>\n

I feel bad that I was unable to participate. I certainly wouldn’t want my actions to harm the career opportunities of another but could no longer bring myself to play the game. Others may feel differently. It was frustrating that the university in question did not want to talk about it.<\/p>\n

But perhaps things are about to take a turn for the better? Today sees the publication of the San Francisco Declaration on Research Assessment<\/a>, a document initiated by the American Society for Cell Biology (ASCB) and pulled together with a group of editors and publishers.<\/p>\n

\"Logo<\/a><\/p>\n

The declaration, which has already been signed by over 75 institutions and 150 senior figures in science and scientific publishing, specifically addresses the problem of evaluating the output of scientific research, highlights the mis-use of impact factors as the central problem in this process and explicitly disavows the use of impact factors<\/em>. I can hardly believe it. This is the research community, in its broadest sense, taking proper responsibility for how we conduct our affairs. I sincerely hope the declaration becomes a landmark document.<\/p>\n

All signatories, whether they be funding agencies, institutions, publishers, organisations that supply metrics or individual researchers, commit themselves to avoiding the use of impact factors as a measure of the quality of published work and to finding alternative and transparent means of assessment that are fit for purpose.<\/p>\n

The declaration has 18 recommendations \u2014 targeted at the different constituencies. The first one establishes its over-riding objective:<\/p>\n

“Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist\u2019s contributions, or in hiring, promotion, or funding decisions.”<\/p><\/blockquote>\n

The remainder go into more detail about what each of the different players in the business of science might do to escape the deadening traction of impact factors and develop fairer and more accurate processes of assessment. By no means does this spell the end of ardent competition between scientists for resources and glory. But it might just be a step towards means of evaluation that are not \u2014 how shall I put it? \u2014 statistically illiterate<\/a>.<\/p>\n

I urge you to download this document (available as a PDF<\/a>) read it and circulate it to your colleagues, your peers, your superiors and those junior to you. Tell everyone.<\/p>\n

And of course, you should sign it<\/a>.<\/p>\n

 <\/p>\n

Update 17th May, 18:28 \u2014<\/strong> I have been discussing my decision \u2014 mentioned above \u2014 not<\/em> to participate in the review of a promotion candidate over at Drug Monkey’s blog<\/a>. He is very critical of my stance and I think may have a point (see his comment thread for details). As a result, while I have not changed my view of the reliance that the selection procedure at the institution involved places on the journal names, \u00a0I emailed them this morning to offer my services as a reviewer (their deadline has not yet passed). I also pointed\u00a0out this blogpost\u00a0and Drug Monkey’s reply by way of explanation but also with a view to pursuing a discussion about their selection process.\u00a0If they take me up on my offer, I think I can provide a review and<\/em> incorporate into it my concerns about the implicit reliance on journal impact factors.<\/p>\n","protected":false},"excerpt":{"rendered":"

Regulars at this blog will be familiar with the dim view that I have of impact factors, in particular their mis-appropriation for the evaluation of individual researchers and their work. I have argued for their elimination, in part because they … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[152,6],"tags":[180,145,254],"class_list":["post-2284","post","type-post","status-publish","format-standard","hentry","category-open-access-2","category-science","tag-impact-factors","tag-open-access","tag-reform"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2284","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/comments?post=2284"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2284\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/media?parent=2284"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/categories?post=2284"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/tags?post=2284"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}