The REF is much in the news, with some feathers ruffled by UK Day One’s proposal to simplify the whole process, as detailed in their report Replacing the Research Excellence Framework. I am sure there are academics and administrators up and down the land who would welcome simplification, but not at any cost, metaphorical or otherwise. Ben Johnson, former Government advisor, has written enthusiastically about UK Day One’s plans but, as James Wilsdon has spelled out, simply relying on the totality of funds earned to drive the distribution of Government funding has all kinds of issues underlying it.
Indeed, Wilsdon was the lead author of the important report The Metric Tide (2015), which considered how the REF could be scrapped, something Prime Minister Gordon Brown had wanted back in 2006, and a simple ‘basket of metrics’ used in its place. This would have simplified things, although I suspect metrics beyond the single figure of already awarded grant income was always envisaged back then, but the response of the sector at the time was deeply negative. Metrics, one or many, remove any opportunity of nuance due to circumstances at any level from the individual to the institutional. The Metric Tide spelled this out in great detail. (It won’t surprise anyone that one of the objections I personally would raise is the obvious statistic about how diversity would be negatively impacted, as was made clear by Wilsdon and co-authors.)
When I chaired the REF Interdisciplinary Advisory Panel, early in the cycle for the 2021 exercise, we were asked to consider whether any of a number of metrics could possibly be helpful in assessing work that crossed boundaries. We went through the list provided one by one and unanimously concluded that no, none of them was likely to be a fit measure. And so none was used.
It is tempting to think that using someone else’s decision about what is ‘excellent’, as would be the case if funding were the sole criterion, would be a sensible choice. However, in practice it simply amounts to outsourcing a decision to other, non-calibrated bodies. Even if one stuck with UKRI funding and assumed that all successful UKRI grants were equally excellent, I fear the evidence would not support that quantification. Let me explain why, directly from my own experience.
Back in the ‘olden days’, before UKRI or even BBSRSC was a thing, I sat on a panel of one of BBSRC’s predecessors, the AFRC (the Agriculture and Food Research Council). At that time grant-holders had to write a final report saying what they had achieved with the money they had been given, and we had to assess them. ‘We’ in this case being the same people (since these were standing committees, which met at regular intervals) as had judged the grant good for funding in the first place. It was dismal to see how many grants we must have raved about three or more years previously were then graded poorly once we read the final reports. Now, of course some failures are to be expected if exciting but risky stuff is to be supported, but too often the outcomes just seemed boring, incremental or non-existent. It made for sobering reflections, as we tried to work out why we got things so wrong.
Some of this may have been hype in the original proposal, promising the moon and we were too naïve to see through this. Some of it would undoubtedly have been due to circumstances beyond their control. I can well remember a (BBSRC) grant of mine which got precisely nowhere, largely because the amount of time the central instrument was functioning properly was so limited. We – by which I of course mean the poor postdoc – did our best, and they didn’t waste their time as they tried to work out where the problems lay when the manufacturers weren’t particularly helpful. But, as far as I recall, we only managed to write one paper and that not of a very high quality. These things happen. But too often, reading the final reports it just felt as if the grant-holders had either lost interest or been buried in other tasks so as to be unable to drive the specific research programme with adequate attention. (I may say I don’t believe our assessments made any difference to anything, although obviously in principle bad final reports could have been used to blackball a particular researcher for some time.)
That is clearly a single data point, but highlights with the best will in the world panels making judgements will sometimes get it wrong. The process is inevitably flawed and to allow further money to flow based on it would just exacerbate inequalities. It really is surely better to judge outputs, and judge them with a human eye. However, beyond that rather major problem, it is worth thinking about unintended consequences if this metric replaced the REF. If grant income is what matters, then more people will be writing more grants, and will feel themselves under pressure to write more grants. More panel members will need to be found. More personnel at the funders to administer the proposals and more administrators in the universities to do the costings… and so it goes on. The costs labelled ‘REF may disappear, the bureaucracy that we all recognize now may disappear, but it will turn up elsewhere to service this different ‘simplified’ approach. That really isn’t a solution.
But there is another fundamental problem that even the ongoing REF is still wrestling with. What price excellence for the people working on grants? Just as professors will feel themselves under more pressure to write more grants, once they have been awarded one that pressure will likely get transferred to the researchers on the grant to produce results at speed to make it easier to obtain further funding. There will be more emphasis on more papers, more hours at the bench, more competition to get that Nature paper out swiftly, and less room for work-life balance, compassion and support as people try to find their way through the academic maze. The research culture element is still being argued over, but as a community we should not allow excellence to be interpreted only with regard to outputs and not take people into account. I fear a funding metric would be intensely detrimental to our lab culture, just when people are beginning to take it seriously. We do not need another generation looking just like the present generation, learning how to compete, bully or fail to bring out the best in their teams. It really is a depressing thought.
There may well be a better way of analysing research excellence, but relying on a crude single number is not likely to be it.
A massive problem in the measuring of proposals for funding is the poor criteria used. When you have had to judge ‘fit to remit’ on a scale from 1-10 where each of the grades 1-10 spans from nationally competitive, internationally competitive to the infamous ‘world class’ it really erodes your trust in any of these processes. When there is no signal to be measured, the only thing you end up measuring is bias. I’d rather we made funding decision at random than primarily on bias – because at least randomization should spread the funding more equitably.
I’ve never seen anyone write a criteria for the excellence of research that I think is even partly functional. Most boil down to ‘well we all recognize excellence when we see it’, which is just another way of saying research done the way I think it should be done, reported in the way it has always been done, IS better research. It just bakes in historic biases and ways of working, with no reference to ‘quality’ of the research at all.
In my opinion attempts to measure research excellence are an entirely flawed endeavour undermined by the simple truth that research is either an acurate representation of what was discovered (in which case it has value) or fraudulent (in which case it does not). Anything outside this binary is primarily measuring guess work and bias.