Metrics, Fulfilment and Career Trajectories

“More effort should be done on understanding people paths. We are too much focused on processes and structures.” JP Bourguignon #esof2014

I was struck by this quote I saw on Twitter from the ERC President Jean-Pierre Bourguignon, because of course we don’t do this at all well. Too often at a departmental level we look at people almost as a commodity, there to deliver excellence – in teaching and research – rather than as individuals to support and allow to flourish.

I assume Bourguignon was discussing the value of ERC Starter grants as a means of giving an initial boost to those outstanding researchers just starting on their personal, independent research trajectories. The success rate for all the ERC grants is only around 10%, many are inevitably going to end up disappointed, but departments need to offer all the support they can to those setting out to find the right way forward for them. Not everyone needs to be a research superstar and not everyone travels in a straight line zooming up through the hierarchy. And all these other people’s aspirations and strengths should be taken into consideration, whether someone is destined to stay in academia or not. We need scientists in the big bad world just as much as in the corridors of academe. And we need to find better ways of assessing who might be best suited to staying in academia which don’t simply rely on outdated ideas and crude metrics of dubious value.

I recently came across an article in Science*, written nearly a year ago by Sandra Schmid who heads up the Department of Cell Biology at the University of Texas Southwestern Medical Center, explaining how they were going to change the way they carried out recruitment in their department. Instead of simply relying on volume of publications or impact factor of journals in which these had been published – in other words those most simple of metrics which removes the need for much thought or judgement – this particular department was going to go for a more subtle approach. This will be an approach that also downplays other aspects that all too readily are used as a proxy for looking at the individual themselves, such as whether the person had been trained in some big name’s group. Instead, a letter covering the following four aspects was to be the submitted material at the first stage (i.e. before interview):

• their most significant scientific accomplishment as a graduate student;
• their most significant scientific accomplishment as a postdoc;
• their overall goals/vision for a research program at our institution; and
• the experience and qualifications that make them particularly well-suited to achieve those goals.

I can see the attraction in this approach because it weakens the power of the purely metrics-driven judgement and takes individual circumstances better into account. It may also reduce the tendency to hype or go for a hard sell that some people find so much easier than others. It should, in principle, allow someone to explain how a particularly innovative approach to some problem has yet to lead to multiple Nature papers (but perhaps they’re on the way), or what they learned from some experiment that failed. Whether it actually leads to ‘better’ hiring decisions will, of course, remain to be seen but it may facilitate hiring individuals who are not just clones of those already in post.

BIS/HEFCE’s call for submissions regarding the use of metrics in any form in assessment has just closed. I am unclear whether a digest of these is actually going to be used to inform REF2020, other aspects of BIS funding or simply as background insight into how quantitative data could or should be used. Whatever, it is probably helpful that the community’s collective views (and I suspect unease) are collated. I am, however, nervous that what a university may say in such a response may not exactly tally with how individuals in that same university actually go about making decisions themselves. Individuals and organisations who have signed up to DORA (Declaration of Research Assessment), for instance, rejecting the use of journal impact factors as a judging criterion, may still implicitly be using them (possibly even barely consciously using them) when it comes to decision-making. If I write, as I just did quite deliberately above, a reference to Nature papers you all know what I mean. I am using Nature as a shortcut to express a ‘good’ paper and in a post like this that’s not a problem. But if I continue to scan an application – be it as a CV or in a more thoughtful, extended cover letter – to look for demonstration of publication in such a journal I am not living up to my signature on DORA. I fear many are still too wedded to such actions and, as with any kind of unconscious bias, we have to bring it explicitly into the open.

So, to return to people paths, what can we all do in our institutions to get away from the mere ‘processes and structures’ that Jean-Pierre Bourguignon referred to? We have to look at each student or postdoc as an individual, with their own strengths and weaknesses. Firstly we need to recognize that some people may have projects that go badly through no fault of their own (bad choice by the supervisor included, but also failure of equipment or delayed supply of samples). If they can explain why, explain what they did to circumvent their problems or how they tackled finding an alternative project to get their teeth into, that may actually be a better recommendation than someone who had an exciting but ‘easy’ set of experiments handed to them on a plate because someone else previously had done all the hard legwork.

Additionally, not everyone will simply travel linearly but some may take time out for all kinds of personal reasons, or decide to move in and out of university research. But we also have to recognize that at every stage people need support, encouragement, advice and, sometimes, a good kicking if they are to fulfil their potential. And finally, we have to appreciate that fulfilling that potential does not always mean sticking around in academia in the vain hope that a job will come up. Those of us sitting in academia will probably only be making decisions about jobs in academia (clearly the thrust of this post) but that’s a tiny part of the world. There are so many places where smart scientists can deliver novel ideas and academia is just one small part of the ecosystem our multitalented early career scientists should be exploring.

*Incorrectly cited as Nature when first posted, corrected (thanks to eagle-eyed Stephen Curry) an hour later.