Peering at review

The House of Commons Select Committee on Science & Technology have announced that they will conduct an inquiry into peer review. They list eight points starting with:

the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

and ending with:

the impact of IT and greater use of online resources on the peer review process, and possible alternatives to peer review.

This is not the first time the committee has looked at publishing. Back in 2004 it produced a report on open access called Free for all? That report was a reasonable round-up of the then state of play but did not precipitate any major changes. It gave a greenish light to UK Research Councils to define their initial, rather weak, policy on OA.

The 2004 report did have something to say on peer review, coming down in its favour and concluding:

As is the case with any process, peer review is not an infallible system and to a large extent depends on the integrity and competence of the people involved and the degree of editorial oversight and quality assurance of the peer review process itself. Nonetheless we are satisfied that publishers are taking reasonable measures to main high standards of peer review. Peer review is an issue of considerable importance and complexity and the Committee plans to pursue it in more detail in a future inquiry.

Well, seven years later, here is that future inquiry.

I had a feeling that POST, the Parliamentary Office for Science & Technology, had produced one of their useful briefing papers on the subject of peer review but was surprised to discover that it was back in 2002, which seems an age ago.

Peer review has been on everyone’s minds recently.  I first started to take notice of peer review in 1999 when Harold Varmus published his E-biomed proposal.  Varmus was head of the NIH at that time, which had recently (1997) launched PubMed as a free index to biomedical scholarly literature. Advised by David Lipman, the genius behind PubMed, and Pat Brown, a leading biomedical scientist from Stanford University, Varmus proposed a wholesale change to biomedical scholarly publishing.

I remember reading this proposal with a sense of disbelief. It did make sense but seemed to ignore the achievements and existing role of publishers (perhaps intentionally?). It proposed a really radical change to the system for publishing research without considering the practical realities of how to achieve that. The system envisaged was summarised thus:

(i) Many reports would be submitted to editorial boards. These boards could be identical to those that represent current print journals or they might be composed of members of scientific societies or other groups approved by the E-biomed Governing Board.

(ii) Other reports would be posted immediately in the E-biomed repository, prior to any conventional peer review, after passing a simple screen for appropriateness.

The proposal attracted some criticism, particularly what it had to say about peer-review.  Stevan Harnad welcomed the proposal but stated:

there is no need whatsoever to tamper with this proven system of  quality control in order to achieve the optimal outcome

Harnad has repeatedly insisted that peer review changes are not necessary to furthering the cause of open access.

The proposal was never realised in its most radical form but as a direct result of it were born the open access publisher Public Library of Science and the NIH repository PubMedCentral. Peer review has had a few tweaks and mini-experiments but I think it has not been seriously threatened since then.

The system that Varmus had proposed back in 1999 still seems radical. Cameron Neylon has recently suggested something similar and discovered that publishing without peer review is far from being a mainstream idea, (though Nature Precedings manages to do it). I attended a talk that Cameron gave last October in the Research Information Network series on Research Information in Transition. He asked what an ideal scholarly communications system for today would look like and stated that it needed to address archiving, registration and communication.  He pointed out that the present system had its origins in the 17th century, in an age of paper and of  centralised production and distribution. It may not be ideal for today with electronic communication and a much more diverse set of published objects.

Cameron suggested that peer review in publishing is too blunt an instrument.  I don’t know – I do hear scientists complaining about peer review in practice, but also hear strong support for the existing system from other scientists.  I think there is a good deal of worry that attempting a change will wreak havoc on the whole academic enterprise and research careers. Major changes could certainly damage the commercial concerns of journal publishers.  What is not clear to me is how significant are any problems with peer review, and whether there is a workable alternative.

I hope the Select Committee inquiry will help to move us closer to finding answers to those questions.

About Frank Norman

I am a librarian in a biomedical research institute. I've been around a few years, long enough to know that exciting new things fall into the same familiar patterns. I'm interested in navigating a path for libraries as we move further from print to electronic resources to open research, and become more embedded in research workflows.
This entry was posted in Journal publishing, Open Access. Bookmark the permalink.

8 Responses to Peering at review

  1. Pingback: Tweets that mention Peering at review | Trading Knowledge -- Topsy.com

  2. THE PURPOSE OF OPEN ACCESS IS TO FREE PEER-REVIEWED RESEARCH FROM ACCESS-TOLLS — NOT TO FREE PEER-REVIEWED RESEARCH FROM PEER REVIEW

    Worth giving this a little more advance thought:

    Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

    Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62.

    Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118. h

    Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292.

    Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242.

    Harnad, S. (1999) Free at Last: The Future of Peer-Reviewed Journals. D-Lib Magazine 5(12) December 1999 

    Harnad, S. (2009) The PostGutenberg Open Access Journal. In: Cope, B. & Phillips, A (Eds.) The Future of the Academic Journal. Chandos.

    Harnad, S. (2010) No-Fault Peer Review Charges: The Price of Selectivity Need Not Be Access Denied or Delayed. D-Lib Magazine 16 (7/8).

  3. Frank says:

    Stevan -yes, quite, as I say above “peer review changes are not necessary to furthering the cause of open access”.

    But there is a quite separate question of whether the current system of scholarly publishing and peer review is the best that can be devised. That is what I think the Select Committee will examine.

  4. I recommend that any scientific paper that is published lists the names of the peer-reviewers that agreed that the manuscript should be accepted, and the posted electronic version include a comments sections so that the readers can provide additional input into the merits and deficiencies in the scientific work.

  5. Do you think any of the possible changes being proposed would actually help the issue over publishing interdisciplinary work I alluded to in my recent post elsewhere on Occam’s Typewriter?

  6. Frank says:

    Athene, I think these problems are certainly on the agenda. Cameron Neylon made a similar point in his comments reported in the Times Higher last month. I think it is helpful to enumerate where peer review is failing and where it is working well, as a prelude to considering whether change is needed.

  7. Pingback: A structured procrastination | Code for Life

  8. Pingback: Peer review inquiry – written evidence | Trading Knowledge