Metrics, Fulfilment and Career Trajectories

“More effort should be done on understanding people paths. We are too much focused on processes and structures.” JP Bourguignon #esof2014

I was struck by this quote I saw on Twitter from the ERC President Jean-Pierre Bourguignon, because of course we don’t do this at all well. Too often at a departmental level we look at people almost as a commodity, there to deliver excellence – in teaching and research – rather than as individuals to support and allow to flourish.

I assume Bourguignon was discussing the value of ERC Starter grants as a means of giving an initial boost to those outstanding researchers just starting on their personal, independent research trajectories. The success rate for all the ERC grants is only around 10%, many are inevitably going to end up disappointed, but departments need to offer all the support they can to those setting out to find the right way forward for them. Not everyone needs to be a research superstar and not everyone travels in a straight line zooming up through the hierarchy. And all these other people’s aspirations and strengths should be taken into consideration, whether someone is destined to stay in academia or not. We need scientists in the big bad world just as much as in the corridors of academe. And we need to find better ways of assessing who might be best suited to staying in academia which don’t simply rely on outdated ideas and crude metrics of dubious value.

I recently came across an article in Science*, written nearly a year ago by Sandra Schmid who heads up the Department of Cell Biology at the University of Texas Southwestern Medical Center, explaining how they were going to change the way they carried out recruitment in their department. Instead of simply relying on volume of publications or impact factor of journals in which these had been published – in other words those most simple of metrics which removes the need for much thought or judgement – this particular department was going to go for a more subtle approach. This will be an approach that also downplays other aspects that all too readily are used as a proxy for looking at the individual themselves, such as whether the person had been trained in some big name’s group. Instead, a letter covering the following four aspects was to be the submitted material at the first stage (i.e. before interview):

  • their most significant scientific accomplishment as a graduate student;
  • their most significant scientific accomplishment as a postdoc;
  • their overall goals/vision for a research program at our institution; and
  • the experience and qualifications that make them particularly well-suited to achieve those goals.

I can see the attraction in this approach because it weakens the power of the purely metrics-driven judgement and takes individual circumstances better into account. It may also reduce the tendency to hype or go for a hard sell that some people find so much easier than others. It should, in principle, allow someone to explain how a particularly innovative approach to some problem has yet to lead to multiple Nature papers (but perhaps they’re on the way), or what they learned from some experiment that failed. Whether it actually leads to ‘better’ hiring decisions will, of course, remain to be seen but it may facilitate hiring individuals who are not just clones of those already in post.

BIS/HEFCE’s call for submissions regarding the use of metrics in any form in assessment has just closed. I am unclear whether a digest of these is actually going to be used to inform REF2020, other aspects of BIS funding or simply as background insight into how quantitative data could or should be used. Whatever, it is probably helpful that the community’s collective views (and I suspect unease) are collated. I am, however, nervous that what a university may say in such a response may not exactly tally with how individuals in that same university actually go about making decisions themselves. Individuals and organisations who have signed up to DORA (Declaration of Research Assessment), for instance, rejecting the use of journal impact factors as a judging criterion, may still implicitly be using them (possibly even barely consciously using them) when it comes to decision-making. If I write, as I just did quite deliberately above, a reference to Nature papers you all know what I mean. I am using Nature as a shortcut to express a ‘good’ paper and in a post like this that’s not a problem. But if I continue to scan an application – be it as a CV or in a more thoughtful, extended cover letter – to look for demonstration of publication in such a journal I am not living up to my signature on DORA. I fear many are still too wedded to such actions and, as with any kind of unconscious bias, we have to bring it explicitly into the open.

So, to return to people paths, what can we all do in our institutions to get away from the mere ‘processes and structures’ that Jean-Pierre Bourguignon referred to? We have to look at each student or postdoc as an individual, with their own strengths and weaknesses. Firstly we need to recognize that some people may have projects that go badly through no fault of their own (bad choice by the supervisor included, but also failure of equipment or delayed supply of samples). If they can explain why, explain what they did to circumvent their problems or how they tackled finding an alternative project to get their teeth into, that may actually be a better recommendation than someone who had an exciting but ‘easy’ set of experiments handed to them on a plate because someone else previously had done all the hard legwork.

Additionally, not everyone will simply travel linearly but some may take time out for all kinds of personal reasons, or decide to move in and out of university research. But we also have to recognize that at every stage people need support, encouragement, advice and, sometimes, a good kicking if they are to fulfil their potential. And finally, we have to appreciate that fulfilling that potential does not always mean sticking around in academia in the vain hope that a job will come up. Those of us sitting in academia will probably only be making decisions about jobs in academia (clearly the thrust of this post) but that’s a tiny part of the world. There are so many places where smart scientists can deliver novel ideas and academia is just one small part of the ecosystem our multitalented early career scientists should be exploring.

*Incorrectly cited as Nature when first posted, corrected (thanks to eagle-eyed Stephen Curry) an hour later.

 

This entry was posted in Academia, Careers and tagged , , . Bookmark the permalink.

4 Responses to Metrics, Fulfilment and Career Trajectories

  1. James Wilsdon says:

    Thanks Athene for another thoughtful and interesting post. To respond briefly to your query about the review of metrics in research assessment, it’s still early days in that process, but I think it’s likely that we will make recommendations against three different time horizons:
    1. Use (and abuse) of metrics now by HEIs and research funders (i.e. before the next REF);
    2. Further uses (or not) of metrics in REF 2020;
    3. Longer-term opportunities, uncertainties and dilemmas in the use of metrics (i.e. areas we want HEFCE and others to monitor closely beyond the next REF).

    As you know, our call for evidence closed last week, and we’re currently digesting the almost 200 submissions received from across the research community. It’s great to have so many responses, and we’ll be taking all of them very seriously as we move to the next stage of the review. I intend to write a piece for the Guardian science policy blog soon, reviewing the balance of evidence received and outlining plans for the rest of the review process. These will include a series of open workshops on key themes (including equality & diversity aspects), which we plan to hold over the autumn. More details to follow soon!

    James Wilsdon (Chair, Independent review of the role of metrics in research assessment)

  2. Laura says:

    Terrible! Women show that they can do well on objective metrics, then oops! suddenly the metrics don’t count any more…

    Metrics are far from perfect measures of scientific quality, but they are still the best way of minimized the effect of (intentional or unintentional) prejudice. ‘Holistic assessment’ sounds like a great way to minimize or downplay achievements of people you don’t really think deserve to achieve. I think that trying to cope with such an environment is much more likely to distort quality science than even somewhat stupid publication and other metrics.

    The University of Texas approach sounds terrible! That’s a set up for the bigwigs on the search committee to pick a candidate based on their personal preferences. Nor does it seem likely to lead to better (or even good) science. With all the system’s (many) flaws, that’s peer reviewed research results that impact the work of other researchers — not charming essays about accomplishments and goals.

  3. Full Professor of Frustration says:

    I”ve sat on over a dozen tenure-track hiring committees – so many I can’t remember. This is where you see the worst of your colleagues. For example on one occasion we were hiring 2 tenure track positions and for the first meeting, the chair decided money was to be a heavily favored metric. This supported his chosen candidate. The next meeting for a different position, money was completely argued out of the metrics because the top person for grant success was a woman that he (and others) didn’t want to hire, because she got in the way of their “pal” that they wanted to hire! When I and others tried to point out this hypocrisy we were utterly ignored and shut down. The shenanigans that go on behind closed doors for hiring decisions are really disgusting. If you have even one bully present, especially if they are a position of power such as department chair, they can wreck havoc on the entire department. And their decisions then snowball because they can hire more bullies or people who will support their bad behavior. Sadly, where I work, upper administrators usually find it easier to support the bully rather than deal with the situation.

  4. Readers may be interested in this account by Jennifer Saul of how the department she heads changed its manner of recruitment to try to remove (or at least reduce) unconscious bias during the process.