Bibliometrics have been making me cross recently.
In the past month, I’ve stumbled across two instances where journal impact factors were being used in a grossly inappropriate way to assess the worth and quality of scientist colleagues. This exposure in turn has really hammered home the inanity of our profession’s obsession with measuring the immeasurable.
I don’t want to compromise anyone’s privacy, so let’s call the two people involved Timothy and Anna (not their real names). Timothy is an early-career researcher in another London university who went to speak to the person in charge of marshalling the troops for the upcoming Research Excellence Framework (REF), a national exercise used to assess the quality of a university’s research output. The better the research, the more money the university gets allocated in the future. So it’s an incredibly big deal that has grown into a monstrously complex bureaucratic nightmare spanning years and consuming many man-hours of effort. Timothy had a few great papers under his belt and was eager to make his contribution. The person he went to see, however, took a quick look at CV and said that it didn’t appear that any of his papers were of a high-enough quality. Timothy was surprised to hear this, and asked how she could tell this just by scanning her eyes down a CV for approximately sixty seconds.
“The journals you’ve published these in,” she explained, “aren’t four-star. So I can tell you right now that we won’t be using your papers in our final REF return.”
Timothy was so shocked that he couldn’t think of anything to say to this; he just took his CV back and retreated, flushed with humiliation. Later though, when I ran into him on the street, he was starting to get angry. I offered to buy him a drink, over which he told me the whole story. What, he wanted to know, did “four-star” mean, anyway? Two of his papers were in the most prestigious and well-regarded specialist journal in his field. Was “four-star” really just code for Science, Nature, Cell or one of their high-impact sister journals? Could it really be that only papers published in these journals were worthy of note? I told him that similar assessments had been made about people at my university, and I’d seen on Twitter that such practices were widespread across the UK.
The next day, Timothy fired off an email to someone higher up at the university who was coordinating the REF and asked if there was an official list of journals ranked by “star”, and if so could he have a copy for reference? That person wrote back immediately, saying that there was no ranking list of journals, as HEFCE are “adamant” that journal impact factors or such similar rankings will not be used by the assessment panels in REF.
Curious, I looked at the REF website and easily found the actual clause that makes this clear:
No sub-panel will make any use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs. An underpinning principle of the REF is that all types of research and all forms of research outputs across all disciplines shall be assessed on a fair and equal basis.
Clearly an injustice had been done. Timothy didn’t know what to do, so I urged him to speak to an eminent professor in his field and get a second opinion about the quality of his papers. I’m happy to report that this divine intervention did the trick, and some of Timothy’s papers were judged to be excellent and worthy of including in the REF return after all. But I wonder how many others are being unfairly judged, and what will happen to them if they don’t have the courage to complain? Although not being included in the REF looks bad for the individual, rocking the boat is not always something that people feel safe doing – especially early-career researchers whose own positions are not yet secure.
Anna is another friend of mine, an early-career researcher at a different Russell Group university further north in Britain. She’s on an independent fellowship at the moment, but this expires in one year and she’s been on the lookout for a permanent position within the same department. When such a job was advertised, she jumped at the chance to try to win the post in a robust competitive process. She ticked every singe box that the job advert wanted: great papers, both in quality and quantity; experience with intellectual property; evidence of bringing in lots of external funding; and great synergy with the work the rest of the groups were doing. The advert also made a prominent statement that women were particularly encouraged to apply because they were currently under-represented in the department. But she was rather shocked not even to be short-listed. When she made some inquiries, it transpired that the committee had triaged applicants solely on whether they had “big” papers. She was afraid to ask what “big” meant, but assumed that it referred to our old friends, the Cell/Science/Nature trinity. Her application had been packed with amazing achievements, but apparently nobody had even bothered to get beyond the publication list.
Much has been said about how impact factors are not a good judge of the individual paper, let alone of the author who wrote it. Recently my fellow Occam’s Typewriter Stephen Curry posted an excellent piece, from which I’ll extract one salient nugget that nicely summarizes the problem (but do read the whole piece, as it’s wonderful):
Analysis by Per Seglen in 1992 showed that typically only 15% of the papers in a journal account for half the total citations. Therefore only this minority of the articles has more than the average number of citations denoted by the journal impact factor. Take a moment to think about what that means: the vast majority of the journal’s papers — fully 85% — have fewer citations than the average. The impact factor is a statistically indefensible indicator of journal performance; it flatters to deceive, distributing credit that has been earned by only a small fraction of its published papers.
As Stephen’s post goes on to point out, the case for impact factors being useful for judging individuals is even more ludicrous.
None of this stuff is new, but this is perhaps the first time that I’ve seen at close hand the human cost of inane bibliometrics. I don’t think my friends are rare exceptions – this cancer is well and truly entrenched.
But as the nights are drawing in and a damp chill settles down over dark London streets, I don’t want to end this post on a negative note. With all this impact factor context in mind, you can imagine my amusement when I received an email from Amazon this morning, telling me that they had come up with a new metric to rank their authors based on subgenre. So I was tickled to discover the following:
Hot damn, I’m officially a Romantic Suspense novelist! I don’t know about you, but considering how many books are for sale overall on this planet, that number looks pretty damned good to me. I may not have a Nature paper yet, but this will do nicely in the meantime.
Post-script: If you’re a scientist getting ground down by the constant harmful vibes of being measured by inappropriate numbers, why not consider escaping into a little bit of “romantic suspense” yourself? The Honest Look and Experimental Heart, my two novels about scientists, are available on Amazon. The first is more literary, the second more hard-core geeky, but both promise chills, thrills, steaming test tubes… and satisfying lashings of lab skulduggery.