Friends with Benefits

A recent study shows – in Switzerland at least – that nominated referees judge grants more favourably than those unconnected with the applicant. I’m afraid I didn’t find the conclusion of the study a surprise. Additionally I suspect that having ‘friends’, nominated referees or simply people you know in the field, is a benefit that will inevitably work better for the well-connected. Well-connected perhaps because they work in a ‘top’ lab, or the applicant is already a giant in the field who knows everyone who is everyone. In this way there is every danger that the Matthew effect will come into play: those already successful will just be given more funding, reinforcing a conservative status quo, hindering the new and original.

Just like the impact factor of a journal not being a good proxy for quality, being already a giant does not necessarily mean that the current grant proposal is well thought through. It is perfectly possible that said PI (Principal Investigator) is operating a grant factory, churning out the equivalent of pot-boilers or ‘outsourcing’ the work to a junior colleague who has not yet mastered the art of grant writing even if there is the germ of a good idea lurking. An objective reviewer, neither swayed by the giant’s reputation, nor alternatively someone who has crossed swords in the past at conferences with this giant or knows their reputation for pot-boiling, should be able to form a more accurate assessment. That is, after all, what objective means. Biases may equally creep in if a long-running feud operates between applicant and referee. This is the reason why it is sometimes permitted by a grant agency for an applicant to name those, individuals or groups, whom they want the application not to be sent to.

I once sat long enough on a grant-giving body (in the days of standing committees) to see a particular applicant fail time after time because one particular referee – this was in the UK and the field was a small one, so this same referee always seemed to be used – raised the same criticism every time, a criticism which was never satisfactorily addressed by the applicant who clearly hoped that somehow they could dodge this referee the next time. In the end the panel got fed up. Feedback was provided that referee and applicant should get together and write a joint grant application to resolve once and for all which of the two was right about how the analysis should be corrected. I think they went on to have a long and productive collaboration.

On the other hand, on this same grant-giving body, I watched with astonishment as the Chair remarked of an application – possibly of a current panel member currently out of the room, but certainly of a friend who had previously sat on the panel (I forget which, but we all knew they were pals) – that of course all of us knew what they had meant to write. Clearly, the Chair agreed, we also appreciated the grant as written wasn’t very good but if funded excellent work would undoubtedly be done. I was gob-smacked, but still quite junior. Luckily another panel member had more courage, or possibly seniority or even both, and called the Chair out for improper behaviour. The grant, I am glad to say, was turned down. I hope the Chair was suitably abashed, although I never enquired.

However that story, which by now is many years old (probably at least twenty) indicates just why the study on using ‘friends’ as referees is important and why we should all be alert to such dangers. Equally, it highlights why I was so pleased to hear Melanie Welham, Excecutive Chair of the BBSRC, say – in our public conversation this week at Churchill College (a recording of which will soon to be on the web) – that ‘safeguarding’ training will soon, possibly already, be required for their panel members, alongside Unconscious Bias training. Time for reflection at the end of a panel, before the rank-order list is finalised, ought to offer an opportunity for any uncomfortable member to speak up; to say that they were worried by the way a grant had been handled, or they felt not enough attention had been focussed on a dodgy referee’s report (perhaps written by someone with an undeclared conflict of interest, i.e. a ‘friend’) perhaps. Such action seems to me to be an excellent idea.

Nevertheless selecting a ‘friend’ as a referee, doesn’t always work. I’ve sat on committees for fellowships where applicants have been able to nominate a referee alongside one chosen by the organisation. It was not that unusual to find the nominated referee expressing strong doubts about the applicant. I wanted to be able to write to the applicant to say ‘choose better next time’, but of course that wasn’t possible. Nominating your referees is the norm when it comes to job applications. Sometimes too, that can go horribly astray. The art, or is it science, of choosing referees probably needs much further thought.

The harsh reality is that, the UK is a small community when it comes to science. Researchers in a field tend to know each other and decisions will have been internalised about their good and bad points by those who’ve been knocking around for a bit. Sometimes it’s hard to remain objective when you know that Professor X treats his students as bench monkeys or that you never agreed with the way they interpreted a certain set of experiments – or alternatively that you think they’re the best thing since sliced bread and think anything they do will yield gold dust. With the best of intentions, it can be hard to forget prior knowledge.  Whereas newcomers may face all the challenges of being comparatively unknown, that sort of baggage will not apply. I know there are those who champion blinding applications as the way forward. But I fear in such a small community it would still be all too easy to guess whose application you were reading, even if personal details were removed.

Related to this point I’d like to highlight the Smith Review into what a domestic fund should look like if EU funding becomes inaccessible to scientists in the UK. As I, along with others, have frequently said, one of the big plus points of the ERC decision-making process is that its panels and referees are drawn very widely from well beyond Europe, and potential conflicts of interest are treated with great seriousness. In considering any new possible future domestic funding, I hope the evidence from the Nature study regarding ‘friends’ and all the benefits they bring, is taken very seriously by the Review group. If a new programme is going, in any faint way, to emulate the ERC we need to be sure refereeing and panels are not peopled by ‘friends’ but by objective experts.