The end of the year is traditionally the time for lists remembering the year just gone: lists of those who died (and, for so-called stars, their marriages and divorces too), events that happened, films that bombed or triumphed, public gaffes or more celebrated quotes. There will also be lists of people\u2019s predictions for the years ahead – whether these be serious or astrological. Lists are a way of organising our lives, our thoughts, and for many of us a way of staying on top of the chaos around us.\u00a0 A quick search through the newspapers came up with these entertaining titles:<\/p>\n
and the inevitable<\/p>\n
Furthermore, today’s latest (web)list of the Top 13 women in Technology<\/a>, features fellow Occcam’s Typewriter blogger Jenny Rohn<\/a>.<\/p>\n I suppose I approach these lists with a rather more personal view this year. I was involved with the production of the Times\u2019 Eureka list of the 100 Most Influential People in British Science earlier this autumn \u2013 this was a rank-ordered list.\u00a0 Deeply dubious about the validity of the whole process, my thoughts on this process were written up previously here<\/a>. (Fellow judge Alice Bell\u2019s thoughts on the meaning of influence can be found here<\/a>, and the actual list \u2013 away from the Times\u2019 paywall \u2013 can be found here<\/a>.)\u00a0 Subsequently and even more absurdly, I found myself appearing on the Telegraph\u2019s list of the 100 Most Powerful Women in Britain<\/a>. This list (which at least had the virtue of only categorizing us and not attempting to draw up a ranking), where it dealt with scientists, had little in common with the women who appeared in the Eureka list, which in itself identifies just how absurd the process is (unless you think the lists ought<\/span> to be subtly different, distinguishing influence from power). \u00a0Mary Beard and I exchanged views, as we wriggled uncomfortably about our appearance in this list in a newspaper neither of us would read under normal circumstances, trying to decide if we were more flattered or annoyed (we concluded we were both). So, from both sides of the judging divide, I think I can confirm the suspicions people are likely to have of these lists \u2013 they are frequently little more than froth designed to waste readers\u2019 time (and extract their money for the media moguls’ pockets) when they should be doing something more useful.<\/p>\n Unfortunately, our world is full of lists that really do matter. Not the ones that appear in the media during the fallow season at the end of a year, but the ones that decree where funding goes, for instance. These include those lists the research councils produce to decide on grants or fellowships, and also the recent lists produced within Government to decide which quangos go, which departments get what swingeing level of cuts. In all cases this is a case of deciding the distribution of pain.\u00a0 So, if the ones in the newspaper are quasi-random, with little hard evidence informing the inclusion of one person and the exclusion of another, are these painful lists any better?\u00a0 Having sat on enough grant- and fellowship-giving panels, it is absolutely clear to me that people try to base decisions on evidence, but it is still hard to come away without a suspicion that it is impossible to form a truly objective view, which gives appropriate weight to appropriate information.\u00a0 There are many reasons for this.<\/p>\n Leaving aside any personally vindictive refereeing (I would actually suspect this is quite rare), there are bound to be referees who have strong views which clash with the proposal they are sent to referee. I recall a particular case from years ago when Dr X submitted a series of grants that were always torn to shreds by Dr Y, and always for the same reason. The research council \u2013 or indeed Dr X \u2013 should have realized this was a hopeless situation much sooner than they did. I think Dr X could have requested (they must have known the identity of Dr Y) that this particular referee not be used \u2013 something research councils allow -, or the research council simply chosen not to use Dr Y. As I recall, the ultimate outcome was a suggestion from the panel that Dr X and Dr Y get together to put in a joint proposal to resolve the argument, which was indeed a fundamental and important one that would otherwise constantly be tossed around.\u00a0 This was only possible because it was a research council with a standing committee (as opposed to the EPSRC\u2019s current system of ad hoc<\/em> committees, where the longevity of the argument would be missed).<\/p>\n A second problem is what happens to grants, or fellowship applications, which are not mainstream or which are inter- or multi-disciplinary.\u00a0 In the latter case, research councils will tell you they share the names of referees, make sure that the referees cover the full spectrum of the proposal, and I don\u2019t doubt this is what they intend to achieve. Unfortunately this leaves the grant at the mercy of referees who may say (for instance of a biological physics grant) that the physics is mundane and they can\u2019t comment on the biology (this from the physicist approached) coupled with a biologist remarking that the biology is not cutting edge and they don\u2019t understand the physics; neither may be capable of judging the originality of the proposal simply because it joins the two strands together to produce something novel and adventurous.\u00a0 Both comments will in essence damn the proposal. \u00a0How can a panel member judge this grant fairly despite everyone\u2019s best intentions? Even, in a less extreme example, if one proposal sits squarely at the centre of a field \u2013 so that each panel member feels they have some idea of what it\u2019s about and why it matters \u2013 and another, with equally good referee reports, is in a less familiar corner of the discipline, how can the panel not collectively feel more comfortable with the former and rank it higher? With the best of intentions, I have seen panels agonise over this and still vote for the familiar versus the strange. They may be right so to do, but it is nevertheless a decision based not solely on merit.<\/p>\n The third problem I would identify is timing of incoming referees\u2019 reports. Anyone who has ever submitted a grant must be familiar with the experience of receiving 2 or 3 favourable reviews, and feeling justifiably smug, only to receive a late and damning response.\u00a0 The problem is that if it is the last report which is negative, there is the danger that the panel will only receive the response to it after the bulk of individual preparation for the panel has been done, or worse as a tabled paper. Even if the referee is out of their depth and talking nonsense (yes, sometimes referees do talk nonsense – which is of course a separate problem, usually because they have refereed at speed and carelessly), the chances are the panel won\u2019t have time to do their homework, read the response thoroughly or check a reference from the rebuttal. They will, however, be left with a lingering feeling of doubt and nervousness.\u00a0 The grant is now damned by timing as much as by ineptitude or lack of originality in the original submission.<\/p>\n At the end of the day, even if peer review is the least bad option, I fear the rank ordering of lists that panels are required to do end up with rough justice due to all these factors. Typically it is clear which applications should sit right at the top, or right at the bottom. But for the rest many factors come into play. Panels try their hardest. I have never seen a grant-giving panel take their responsibilities with anything less than seriousness, never as chair had to take a group to task for not concentrating hard on the business in hand, but that doesn\u2019t mean we get things right. Panels end up being conservative. Despite admonitions that risk is to be encouraged, this can only be up some notional and invisible point and feasibility studies are always encouraged. Members may be unconsciously swayed by factors they should be ignoring (\u2018I met that referee once and (s)he\u2019s a Good Guy –\u00a0 or a Heavyweight – so (s)he must be right in their views\u2019, despite the fact their view contradicts other referees; \u2018I don\u2019t think much of that department\u2019 so they mark down a new investigator there; graphene is the up-and-coming material, so that grant must be better than one studying gallium arsenide\u2026. one could draw up a list in itself of unconscious bias in refereeing!\u00a0 Unconscious bias can turn up in many guises.).\u00a0 There is always the danger, when the chips are down and difficult decisions have to be made, that a throwaway and not-well-thought-through remark from one panel member can seem like a lifeline to others to absolve them from facing up to a difficult choice.<\/p>\n So, just as I said about the Eureka list, there is no single right answer. The difference is that people\u2019s futures hang on the outcomes of grant panels in a way no media list I am likely to encounter can do. As the funding gets tighter, and yet tighter again, the element of lottery in these lists becomes more damaging. The stated intentions of some research councils to \u2018concentrate\u2019 their grant money means that earlier decisions will be reinforced in the future, so that having lost out once researchers may be at more risk of losing out again. I don\u2019t know what the answer is, but I fear these lists are going to be hanging over us for a long time to come.<\/p>\n","protected":false},"excerpt":{"rendered":" The end of the year is traditionally the time for lists remembering the year just gone: lists of those who died (and, for so-called stars, their marriages and divorces too), events that happened, films that bombed or triumphed, public gaffes … Continue reading