(This is a guest blog I wrote for the Research Information Network.)
I’m a fan of peer review.
There, I’ve said it. And I’m not saying it in the way that Sir Winston Churchill famously spoke of democracy; ‘the worst form of government except for all those others that have been tried.’ I’m also not talking as one with no experience of the peer review process, nor one with uniformly good (or bad) experience–I’ve had papers accepted without hesitation, I’ve had others improved substantially by the review process, and I’ve had a manuscript bounced around for two years until we found a sympathetic editor and reviewers who understood just what the bloody hell we were talking about.
I suspect that my experience matches that of the vast majority of jobbing scientists. We don’t have particular axes to grind, we just want to get our stuff published in as ‘good’ a journal as possible (and there’s a whole other can of fish to worry about) and we want to move on to the next experiment. Truth be known, we’d also like to see what our colleagues and, well, peers make of our work, and maybe even make constructive comments before the entire world gets to see it.
I’m also not a football fan of peer review. I won’t support this system unthinkingly, against all comers, waving my blue and white scarf above my head and throwing rolls of toilet paper at the opposition (actually, that attitude seems to characterize most opponents of peer review, but more on them in a bit). No; I recognize there are problems, and I’m certain it could be improved. Just don’t ask me how: I don’t know, for example, if single or double blind review, or complete openness would improve matters or make them worse, all things considered.
The thing is, peer review has been getting a bit of stick recently. Medical journals, especially, seem to get very worked up about it. The matter of Medical Hypotheses is another case in point. The Editor, Bruce G. Charlton, is embroiled in a fight with Elsevier, the publisher. Elsevier wants to make it peer reviewed, and Charlton thinks that will destroy the spirit of Medical Hypotheses. As if one editor is going to be in any way ‘better’ at assessing a manuscript’s ‘worth’ or ‘rightness’ (not newsworthiness–that’s different entirely and it’s something the crew at, say, Nature do very well) than any three peers–and let’s be explicit here, in this sense ‘peers’ means ‘experts in their field’, right? Anyway, I’m not overly concerned about the pros and cons of that case (except to point out that every professional in the field knows exactly how much worth to place on anything published there. It’s only the press–and the naif–that get confused about such things): rather I’d like to look (briefly, because it’s far too painful to spend much time on) at some of the comments on the Nature news article.
First, the repeated assertion that Nature, by virtue of having full-time editors, does not do peer review is patently ludicrous. How anyone who practises science could hold that opinion, moreover repeat it in a very public forum, is beyond me. At least the commenter had the grace to admit he was wrong, but this demonstrates the sheer level of misinformation and ignorance surrounding the entire issue. How can we have a reasonable debate when even those involved don’t know what they are talking about?
Second, we have the frothing-at-the-mouth prophets:
Peer Review should be universally rejected by all scientists and researchers as the thought control experiment that it is. Science is not Consensus! Truth cannot be discovered by vote no mater how smart the electorate thinks it is.
Right. There is so much wrong in those three little sentences that I really don’t know where to begin. Let’s just say that it’s another exemplar of people Just Not Getting It.
It gets better. It always does:
Peer Review as a concept, is no more than an easy way to influence, suppress, and control the direction and funding of scientific research. It is no less than tyranny and must be rejected as such.
Uh, OK, keep taking the tablets–actually no, get some different ones because those aren’t working.
Please, just one more?
Both Socrates and Galileo were “Peer Reviewed”. We all know how well that worked out.
Ah here we are. We’re dealing with someone who has had an obviously quite brilliant Idea that nobody will publish, let alone listen to, and the problem is not the Idea but peer review! If I had a quid for every loon who emailed me his theory on how everybody is wrong and how he (invariably a ‘he’; ‘she’s are far too sensible) has found the secret to life the universe and 42, only he can’t publish it because of the tyranny of peer review, and look how nobody believed Socrates or Galileo or Einstein, oh God especially poor old Einstein; well, I’d have fifteen pounds and sixty pence (that last one was a particularly sad and tragic affair).
Look, people: we’ve moved on from arresting or murdering people for having wacky scientific ideas–at least in the West. There is no conspiracy, no shadowy cabal that stops you publishing anything (and indeed, the internet makes publishing ridiculously easy). It’s far more likely, all things considered, that your Idea is so much dingo juice and you’re suffering a severe case of sour grapes. If your Idea is, actually, good; then Time will prove you right. It always does. But my money is on you being a wacko.
Even the company I work for is not trying to replace the initial round of peer review. Yes, our Chairman jumpstarted the Open Access revolution (and you’ll see why I was keen to establish my own bona fides, above), and yes, we’re really interested in what I’m calling post-publication peer review–assessing the likely impact and importance of the scientific output soon after the point of publication–but even we recognize the value of peer review as it stands today (notwithstanding arguments about anonymity and the like).
Ah! A voice of sanity and reason:
But people whose ideas, popular or not, that are backed up by sloppy research, or no research at all, should not be published until they can come up with the proper evidence supporting their claims. Coming up with adequate supporting evidence is another driving force behind science and ensures its credibility.
The point of peer review, actually, is neither to suppress nor promote good, bad, wacky, conventional, nuclear or world-changing Ideas. The major question that peer review is designed to answer, and is best at answering, is, “Is it done right?” It’s not some vast conspiracy to keep ideas down, nor to deny lunatics a forum or grant money. It’s there to help workaday scientists (some of whom will have brilliant, paradigm-shifting Ideas) do their research, without having to wade through a Stygian morass of ill thought-out crap.
We have to know our limitations, of course. Peer review suffers terribly from poison-pen reviewers. A field I worked in was almost destroyed because one PI kept trashing the community’s papers in review stage (she turned out to be mentally ill; that, also, is a story for another day). But that’s no reason to throw the F1 out with the autoclave bags. We need to identify and fix the problems with peer review, not destroy it entirely.
I need to address one last misconception. Peer review, done properly, might guarantee that work is done correctly and to the best of our ability and best intentions, but it will not tell you if a particular finding is right–that’s the job of other experimenters everywhere; to repeat the experiments and to build on them. Indeed, a friend of mine has been known to say, in public even, that most things in Nature are wrong. (And he should know–he’s a Nature editor.) He’s right of course–everything published will be superseded. But the point is, and the point we’re in danger of losing sight of to our great detriment as jobbing scientists, is that peer review done even half-arsedly cuts out a whole pile of junk and lets us get on with the real business of Science; that of finding shit out.
