Should scientific peer review be thrown on the scrap heap?

Should peer-review be done away with in the sciences?

The assurance of quality through the peer-review process is the sacred cow of modern scientific practice.

Richard Smith thinks its time the cow is taken to the abattoir – that scientists should publish first and then let the scientific community decide afterwards – or as he puts it ‘publish then filter’.

Despite Smith’s alternative view, the peer-review process is essential to all of the sciences.

Peer-review is like democracy, just because it isn’t a perfect process doesn’t mean it should be tossed aside – it’s important. It’s important because it allows someone else to assess, in principle independently, the validity of scientific work. Just because it is not always done well, doesn’t mean it should not be done at all.

Smith began this debate in the British Medical Journal after Cathie Sudlow called into question the practice of peer-review in the same journal .

Her criticism centers on a recent ‘publication’ in Science, the flagship journal of the American Association for the Advancement of Science (AAAS). This study apparently presents evidence suggesting a viral link to the elusive disease Chronic Fatigue syndrome, perhaps giving false hope for many Chronic fatigue sufferers.

Sudlow has criticized the methods used in the Science paper and cited several other studies which have found no causal link between the virus and Chronic Fatigue in her support. As a result, she questions the practices of peer-reviewed scientific journals, especially top-journals such as Science – a reasonable use of babies and bathwater for population reduction – and believes there should be more responsibility from editors and reviewers: that journals shouldn’t rush to publish (even though she wanted them to rush to publish her E-letter which expressed doubt concerning the methods used in the Science article) – and authors themselves to not overstate their results.

The peer-review process is essential to publication in any scientific discipline.  Even though physicists, mathematicians and astrophysicists as Smith claims do, sometimes, pre-publish on the web for comments from the community (such as arXiv) – they ultimately publish in the same way as the other sciences – e.g. through peer-reviewed journalism.

Which is a good thing.

The process is undoubtedly imperfect but it still remains sound practice.  Scientific work, once published, is not set in stone but rather opens up a discourse between scientists – simply put, if you don’t agree with the science published, publish work which calls this into question yourself.  This too is an essential part of the scientific process, which Sudlow herself is actively and appropriately engaging in.

‘Publish then filter’ is not a good answer, at least as Smith depicts it.  In fact this mechanism is already in wide (though perhaps not widely accepted) practice.  Lots of people do already ‘publish then filter’ outside of the scientific literature – and lots of silly non-scientific ‘science’ makes it into the news as both Ben Goldacre and Bob Park have often pointed out. The mad also have fun with random ‘press-science’ such as the law suits to prevent the LHC from running, because it will end the world, and put ninja-seagulls from the future, armed with baguettes, out of business (but then they knew that anyway – they’re from the future (but they are seagulls…..)).

In this way, the ‘publish then filter’ mechanism is already in place, but doesn’t need to be included in the scientific literature itself.  Just because it’s in the news, doesn’t mean its good, or real for that matter, science.

‘Publish-then-filter’ kills one of the main features of peer review in science – the removal of obvious nonsense from the wider community’s scientific discussion.

Setting aside the ‘slash-and-burn’ model of changing the system of scientific evaluation, there are undoubtedly some ways that peer-review and scientific dissemination in general could be improved.

For example, in opposition to the ‘full disclosure’ of authors and referees Sudlow recommends, double-blind peer-review could be implemented, where both the authors (and their institutions) as well as the referees are anonymous.

There are several studies which indicate that this could be effective. For instance, see a studies in the The American Economic Review and in Behavioral Ecology.

Further, as a tool to improve dissemination, institutions such as the AAAS and the Royal Society could have websites or blog posts allocated to certain topics, with a short review, such as the science-writer Philip Ball has on ‘Water in Biology’. This would allow for easier access to work spread throughout many journals on a particular topic.

The peer-review question notwithstanding, there is a large underlying issue to this debate which addresses the culture and practice of modern science. Institutional practices as well as the ethos of the scientific community itself needs to be reassessed. But that is for another day….