Some time ago I had the curious experience of being an observer at a series of decision-making meetings covering different disciplines. As a mere observer I didn’t have to do any preparation myself, none of this producing scores for the applications, or having to do the difficult task of comparing apples and pears. But I could be an observer in all senses, including anthropological, as I watched how the disciplines compared in their approach to a comparable problem and also how individuals tackled the task of analysing the applications. This time, unlike some committees I have been on, everyone had done their homework very thoroughly. However, not all the panel members were as successful in delivering what I think are the key skills at such meetings: being able to summarise the key plusses and minuses in the application in front of the panel and convey the important points to everyone in the room.
What struck me time and time again was the care every panel member had taken in preparing lengthy notes on each application to which they had been asked to speak, but also the lack of thought they then put into the delivery of this material. Many, probably the majority, of the participants read their notes more or less verbatim. This in itself can be very boring, as few scientists of my acquaintance read notes (either for a situation such as this, or when in the different role of giving a lecture) with much energy. It tends to come across as rather boring, whether or not the content is in itself uninteresting. Worse, too often it is read in a soft monotone which does not carry across a room. Many panel meetings are held in large hotel conference suites, usually with lousy acoustics, often with a noisy fan for the air conditioning (in summer), or the constant racket of traffic on a motorway thundering past just outside. Senior academics really should be capable of pitching their voice so that colleagues 10 feet away can hear what is said; disappointingly often it doesn’t seem to cross their minds that this is an important part of their job when sitting on committees. Microphones are rarely available and, as my experience in my own University’s Council Room has shown me, make for a disruptive hassle if used, even though the high-ceiling of such an elegant room can make such use desirable.
At these panel meetings as so often, some crude pre-scoring had been done and a rank ordered list produced. This meant that it was obvious that a certain percentage were never going to make the cut, say 50%. Of course it is important to establish that no errors in transcription of marks into the mark-sheet have occurred and that no one has had second thoughts about their marks. Furthermore it is also necessary to check that no one has obviously been scoring at total odds to everyone else, for instance by giving all top marks or all bottom ones, rather than a reasonable spread across the board; or that one panel member has given a top mark which has been pulled down into the no-hoper range by a completely divergent score from a fellow panel member.
But if none of these caveats apply, then it ought to mean these apparent no-hopers can be dealt with quite briskly. Yet still, too many panel members read out their notes in full, regardless of the fact that a sentence or two in summary is really all that is required to enable feedback to be given, or at the very least for everyone to have confidence that the application really does sit towards the bottom of the score-sheet. I find it very frustrating to have to listen to interminable droning on about an application that is fundamentally flawed on some front or other. If the chair isn’t forceful this can happen only too easily, and cause committee meetings to drag on far longer than should be necessary. This drawing out matters because it can mean that applications discussed late in the day can get short shrift, as everyone starts consulting their watches and their iPhones about their return travel plans.
When I chair such a meeting, I often feel that for a very few applications sitting right at the top of the rank-ordered list, a brief discussion may again be all that is required. If an application is definitely going to be funded/sent out to referee or whatever, why waste time extolling its virtues. Again, a brief summary should be sufficient, yet rarely seems to be regarded as such.
I came away from these meetings impressed by everyone’s diligence, but depressed by the feeling that many of the members were in essence completely unaware the dynamics of what was going on, and had not troubled to think about how to make the meetings effective. Mumbling to yourself as you read out screeds of stuff which no one else wants to hear because it is obvious the application is going to bomb, seems to me to be a cardinal sin. Time and effort should be saved for the tricky middle-ranking cases, where the evidence needs to be carefully weighed up and all relevant information chewed over. And that really does require that everyone in the room hears every syllable of the analyses presented, not simply the odd phrase when the speaker temporarily gets their nose out of their notes sufficiently to address the room audibly. Only in that way can everyone leave the room confident total justice was done; that the outcome was not a lottery due to a particularly persuasive individual winning an argument over a mumbling one’s more reasoned arguments; or that cases presented early in the day fared better (or worse) than those that came later; and finally that any tribal tendencies visible between one part of a discipline and another are scrutinised fairly.
So panel members of today – and tomorrow – please bear the following in mind:
- Speak up: Remember that all your homework is in vain if no one can hear what you say when you speak to a particular case.
- Speak succinctly: People may tune out if you ramble on at unnecessary length when speaking to third-rate applications, and fail to spot the cases where you really feel there is true excellence because they’ve stopped paying attention to your verbose sentences.
- Speak fairly: Do not let personal objections – to an experimental approach, a sub-discipline or to an individual – override the evidence presented in the case before you.
- And equally, listen fairly and do not let a post-prandial dip in energy levels cause you to miss some of the discussion or your own prejudices (including against fellow panel members) distort the message you hear.
All so much easier said than done!
I seem to have spent the best part of 20 years involved in peer review meetings of various forms – involving different disciplines, organisations, meeting processes, personalities and chairs. I couldn’t agree more with your comments. Meetings fail more through unglamorous practicalities, personal communication issues and the social dynamic of the group than on matters of principle.
One observation I would make, is that people often feel the need to “say their piece” not out of regard for themself, but rather though a percieved need to “do justice to the proposal”. This is an honourable attitude – albeit not particularly necessary or efficient in terms of overall goals.
Don’t underestimate the value of good preparation though!
Mark Claydon-Smith
EPSRC
Maybe people could separate speaking for the proposal into initial comments, and more detailed comments. If the panel unanimously agree to skip the detail as the case is already won or lost then skip it. Clearly you would have to do the same preparation in case a proposal you thought would bomb didnt…
Agreed. But as you imply, it is the chairs who bear much of the responsibility, and who should tell panel members *before* the meeting what is expected of them.
I haven’t had the pleasure of meetings like these yet but now I feel thoroughly prepared if I ever do. Doesn’t seem like much to ask and should be obvious to everyone anyway – but that’s always a sign of great advice!
I’ve had my first experience of this kind of meeting recently on the Societies Syndicate grants committee – it’s amazing how much verbiage can be exchanged over issues of a few pounds. Maybe I should forward this blog post around the other committee members…
For some years I was a member of a BBSRC Grant Committee. I served under four different Chairs, all excellent, who ran the meetings very smoothly. There were no written rules but there was a “house style” for speaking about a grant that everyone adopted. Basically you had to be clear and succinct because there were often more than a hundred grants to get through. Nobody waffled and everyone spoke up well. It’s amazing how much a little pressure focuses the mind.
In my experience, the best thing to do is to set a lower cutoff for general discussion, but allow any panel member who wishes to do so to make a case for raising an application from below to above the cut-off. That can be sufficient motivation for cogency.
At the top it’s probably essential that the cases be made – but a good chair can insist on brevity.