The next day, Timothy fired off an email to someone higher up at the university who was coordinating the REF and asked if there was an official list of journals ranked by \u201cstar\u201d, and if so could he have a copy for reference? That person wrote back immediately, saying that there was no ranking list of journals, as HEFCE are \u201cadamant\u201d that journal impact factors or such similar rankings will not<\/em> be used by the assessment panels in REF.<\/p>\n Curious, I looked at the REF website and easily found the actual clause <\/a> that makes this clear:<\/p>\n No sub-panel will make any use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs. An underpinning principle of the REF is that all types of research and all forms of research outputs across all disciplines shall be assessed on a fair and equal basis.<\/p><\/blockquote>\n Clearly an injustice had been done. Timothy didn\u2019t know what to do, so I urged him to speak to an eminent professor in his field and get a second opinion about the quality of his papers. I\u2019m happy to report that this divine intervention did the trick, and some of Timothy\u2019s papers were judged to be excellent and worthy of including in the REF return after all. But I wonder how many others are being unfairly judged, and what will happen to them if they don\u2019t have the courage to complain? Although not being included in the REF looks bad for the individual, rocking the boat is not always something that people feel safe doing \u2013 especially early-career researchers whose own positions are not yet secure.<\/p>\n Anna is another friend of mine, an early-career researcher at a different Russell Group university further north in Britain. She\u2019s on an independent fellowship at the moment, but this expires in one year and she\u2019s been on the lookout for a permanent position within the same department. When such a job was advertised, she jumped at the chance to try to win the post in a robust competitive process. She ticked every singe box that the job advert wanted: great papers, both in quality and quantity; experience with intellectual property; evidence of bringing in lots of external funding; and great synergy with the work the rest of the groups were doing. The advert also made a prominent statement that women were particularly encouraged to apply because they were currently under-represented in the department. But she was rather shocked not even to be short-listed. When she made some inquiries, it transpired that the committee had triaged applicants solely on whether they had \u201cbig\u201d papers. She was afraid to ask what \u201cbig\u201d meant, but assumed that it referred to our old friends, the Cell\/Science\/Nature<\/em> trinity. Her application had been packed with amazing achievements, but apparently nobody had even bothered to get beyond the publication list.<\/p>\n Much has been said about how impact factors are not a good judge of the individual paper, let alone of the author who wrote it. Recently my fellow Occam\u2019s Typewriter Stephen Curry posted an excellent piece<\/a>, from which I\u2019ll extract one salient nugget that nicely summarizes the problem (but do read the whole piece, as it\u2019s wonderful):<\/p>\n Analysis by Per Seglen<\/a> in 1992 showed that typically only 15% of the papers in a journal account for half the total citations. Therefore only this minority of the articles has more than the average number of citations denoted by the journal impact factor. Take a moment to think about what that means: the vast majority of the journal\u2019s papers \u2014 fully 85% \u2014 have fewer citations than the average. The impact factor is a statistically indefensible indicator of journal performance; it flatters to deceive, distributing credit that has been earned by only a small fraction of its published papers.<\/p><\/blockquote>\n As Stephen\u2019s post goes on to point out, the case for impact factors being useful for judging individuals is even more ludicrous.<\/p>\n None of this stuff is new, but this is perhaps the first time that I\u2019ve seen at close hand the human cost of inane bibliometrics. I don\u2019t think my friends are rare exceptions \u2013 this cancer is well and truly entrenched. <\/p>\n But as the nights are drawing in and a damp chill settles down over dark London streets, I don\u2019t want to end this post on a negative note. With all this impact factor context in mind, you can imagine my amusement when I received an email from Amazon this morning, telling me that they had come up with a new metric to rank their authors based on subgenre. So I was tickled to discover the following:<\/p>\n Hot damn, I’m officially a Romantic Suspense novelist! I don\u2019t know about you, but considering how many books are for sale overall on this planet, that number looks pretty damned good to me. I may not have a Nature<\/em> paper yet, but this will do nicely in the meantime. <\/p>\n Post-script:<\/strong> If you\u2019re a scientist getting ground down by the constant harmful vibes of being measured by inappropriate numbers, why not consider escaping into a little bit of \u201cromantic suspense\u201d yourself? The Honest<\/a> Look<\/a> and Experimental<\/a> Heart<\/a>, my two novels about scientists, are available on Amazon. The first is more literary, the second more hard-core geeky, but both promise chills, thrills, steaming test tubes\u2026 and satisfying lashings of lab skulduggery.<\/p>\n","protected":false},"excerpt":{"rendered":" Bibliometrics have been making me cross recently. In the past month, I\u2019ve stumbled across two instances where journal impact factors were being used in a grossly inappropriate way to assess the worth and quality of scientist colleagues. This exposure in … Continue reading ![\"RomanticSuspense\"](\"http:\/\/farm8.staticflickr.com\/7263\/8074684756_143a20dc88.jpg\")
<\/a><\/p>\n