Peering at review

The House of Commons Select Committee on Science & Technology have announced that they will conduct an inquiry into peer review. They list eight points starting with:

the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

and ending with:

the impact of IT and greater use of online resources on the peer review process, and possible alternatives to peer review.

This is not the first time the committee has looked at publishing. Back in 2004 it produced a report on open access called Free for all? That report was a reasonable round-up of the then state of play but did not precipitate any major changes. It gave a greenish light to UK Research Councils to define their initial, rather weak, policy on OA.

The 2004 report did have something to say on peer review, coming down in its favour and concluding:

As is the case with any process, peer review is not an infallible system and to a large extent depends on the integrity and competence of the people involved and the degree of editorial oversight and quality assurance of the peer review process itself. Nonetheless we are satisfied that publishers are taking reasonable measures to main high standards of peer review. Peer review is an issue of considerable importance and complexity and the Committee plans to pursue it in more detail in a future inquiry.

Well, seven years later, here is that future inquiry.

I had a feeling that POST, the Parliamentary Office for Science & Technology, had produced one of their useful briefing papers on the subject of peer review but was surprised to discover that it was back in 2002, which seems an age ago.

Peer review has been on everyone’s minds recently.  I first started to take notice of peer review in 1999 when Harold Varmus published his E-biomed proposal.  Varmus was head of the NIH at that time, which had recently (1997) launched PubMed as a free index to biomedical scholarly literature. Advised by David Lipman, the genius behind PubMed, and Pat Brown, a leading biomedical scientist from Stanford University, Varmus proposed a wholesale change to biomedical scholarly publishing.

I remember reading this proposal with a sense of disbelief. It did make sense but seemed to ignore the achievements and existing role of publishers (perhaps intentionally?). It proposed a really radical change to the system for publishing research without considering the practical realities of how to achieve that. The system envisaged was summarised thus:

(i) Many reports would be submitted to editorial boards. These boards could be identical to those that represent current print journals or they might be composed of members of scientific societies or other groups approved by the E-biomed Governing Board.

(ii) Other reports would be posted immediately in the E-biomed repository, prior to any conventional peer review, after passing a simple screen for appropriateness.

The proposal attracted some criticism, particularly what it had to say about peer-review.  Stevan Harnad welcomed the proposal but stated:

there is no need whatsoever to tamper with this proven system of  quality control in order to achieve the optimal outcome

Harnad has repeatedly insisted that peer review changes are not necessary to furthering the cause of open access.

The proposal was never realised in its most radical form but as a direct result of it were born the open access publisher Public Library of Science and the NIH repository PubMedCentral. Peer review has had a few tweaks and mini-experiments but I think it has not been seriously threatened since then.

The system that Varmus had proposed back in 1999 still seems radical. Cameron Neylon has recently suggested something similar and discovered that publishing without peer review is far from being a mainstream idea, (though Nature Precedings manages to do it). I attended a talk that Cameron gave last October in the Research Information Network series on Research Information in Transition. He asked what an ideal scholarly communications system for today would look like and stated that it needed to address archiving, registration and communication.  He pointed out that the present system had its origins in the 17th century, in an age of paper and of  centralised production and distribution. It may not be ideal for today with electronic communication and a much more diverse set of published objects.

Cameron suggested that peer review in publishing is too blunt an instrument.  I don’t know – I do hear scientists complaining about peer review in practice, but also hear strong support for the existing system from other scientists.  I think there is a good deal of worry that attempting a change will wreak havoc on the whole academic enterprise and research careers. Major changes could certainly damage the commercial concerns of journal publishers.  What is not clear to me is how significant are any problems with peer review, and whether there is a workable alternative.

I hope the Select Committee inquiry will help to move us closer to finding answers to those questions.