On peer review, part 451 (b)

I had a conversation on twitter last night with m’learned friend Nige, who runs the most ethical small business I know of.

He pointed me at this blog post by Richard Smith at the BMJ, What is post publication peer review?.

You know, there are days when I wish I hadn’t used the phrase ‘post-publication peer review’ to describe what F1000 does. It’s inevitably misunderstood. Post-publication peer review, as I intended the phrase, is not and never could be a replacement for peer review ‘proper’ (if you like). What F1000 does is look at papers after they’ve been published and say “Hey guys (and gals), this one’s worth reading. Here’s what it purports to say,” dot dot dot.

Peer review ‘proper’ (if you like) says “Yeah, they did the right stuff here, but there are a couple more experiments they should do to be sure,” or something like that. And if you’re reviewing for a glamour mag, you might also get a comment along the lines of “ARE THEY OUT OF THEIR TINY LITTLE MINDS?” or similar. That, especially the first part (whether the experiments were done right, the controls are appropriate, the relevant literature has been reviewed and cetera, &c., etc.) is not going to happen ‘post-publication’. No way, no how, not ever, uh-uh, over Karl Popper’s dead body [check that Popper is dead—Ed.].

You see, the thing is, the real, incontrovertible and indisputable thing is about science, is that it’s all provisional. (Henry has made this point somewhat forcibly several times in the past, most famously by asserting “Everything Nature publishes is wrong”.) Findings reported in papers only ever become less provisional when somebody repeats the experiment in a different lab and gets the same (or similar) results. (Nige and I discussed clinical trials briefly, which is where the ethics thing came up, but even there you should be able to decide on what’s ‘true’ through meta-analyses and Cochrane Reviews and whatnot.)


That, and only that, is true post-publication peer review.

Not chatting about a paper on blogs or in Nature; not commenting on a manuscript thrown up on a pre-print server; not talking about posters; not even F1000.

So, please, can we drop all this nonsense about doing away with peer review (‘proper’—if you like)? It’s not big, and it’s not clever.

About rpg

Scientist, poet, gadfly
This entry was posted in Literature, Nonsense, The stupid, it burns and tagged , . Bookmark the permalink.

14 Responses to On peer review, part 451 (b)

  1. cromercrox says:

    I’ve been thinking about post-publication peer-review (henceforth PPPR) a lot recently. What its advocates seem to miss is that the process of peer-review isn’t a one-shot deal.

    Please correct me if I’m wrong, but papers are not simply published or rejected after one round of review. This almost never happens, in my experience. In reality, a paper, once selected for peer review by the journal’s board of reviewing editors (whether in house or externally), will go through at least two rounds of review – in extreme cases, as many as four or five – before a final decision is reached. In between there is a great deal of discussion: between editors themselves; between editors and referees; between editors and authors (the task of editors being somewhat like that of the ringmaster in the circus, making sure everything happens in order and that the elephants don’t trample the clowns) — and occasionally between referees and between referees and authors.

    There’s a long and winding road between submission and publication. Most authors (even some whose papers have been rejected) agree that the process, while arduous and occasionally nerve-wracking, has the net effect of improving the paper, making it more accessible, more focused, and – one hopes – more reproducible,.

  2. Peer review may be the best thing we gave but it works only to maintain a hierarchy of journals, The insane pressure to publish has spawned a huge proliferation of journals and the consequence is that any work, however bad, will get published in some journal that claims to be peer reviewed.

    At one extreme, publishers like Elsevier and BMC publish journals that deal with various forms of quackery, They are listed in Pubmed and claim to be peer reviewed (by other quacks of course).

    There is nevertheless a real problem with post-publication peer review. You’ll get cheers from scientific colleagues if you criticise witch-doctery like homeopathy, but if you criticise openly people doing real science, you are likely to get a very frosty reception. Most people won’t do it, but that, I believe must change in the interests of keeping science honest.

  3. Stuart says:

    @cromercrox – I suspect that things work differently here in our ivory towers at Nature HQ compared with many other publishers. When it comes to chemistry, for a lot of journals out there I get the impression that peer review is pretty much a one-shot deal for most manuscripts (I’ve worked for the ACS and know a lot of people who have worked for the RSC) – one round of review and then accept or reject. Maybe the top journals in each field (which is quite a small percentage of all the journals out there) do have more thorough peer-review processes, but I suspect that for the vast majority of papers at a vast majority of journals, it’s a one-shot deal.

  4. rpg says:

    But it’s that hierarchy that allows us to identify the crap, even the crap that claims to be peer-reviewed, surely?

    If you do away with peer review as at least an initial filter of work that has been done according to good practices (note I’m not saying anything about whether the work in question is *right*: that’s the realm of reproducibility) then it’ll be a stinking mess as everybody, including the quacks, publishes EVERYTHING.

    And I suspect, nay predict, that out of such a morass would arise an elite group who peer review and read each other’s papers. And they’ll be doing the real, reproducible, science.

  5. Unfortunately it isn’t that simple, There are plenty of retractions in Nature, and plenty of wrong results. My most highly cited paper is a Nature letter of mind-numbing triviality. The problem arises from the crazy pressure put on people by univerities and research councils to publish a paper every 10 minutes. This naturally gives rise to the publication of half-finished work. Journals like Nature actually add to the problem rather than help it because of the totally irrational weight that is put on publication in them by pinheaded administrators. See for example Challenging the Tyranny of Impact Factors, published in, ahem, Nature, and How to get Good Science.

  6. rpg says:

    While there are wrong results published (I made this point, implicitly) and retractions, that doesn’t mean we should throw out the baby with the bath water. I fail to see how the system the wild-eyed prophets propose is actually going to improve anything, least of all the pressure to publish.

  7. “Everything Nature publishes is wrong”. Indeed.

  8. cromercrox says:

    Of course all scientific results are provisional. That’s what I meant by my statement that everything published in Nature is wrong. To take that statement quite as literally as Michael and others seem to do betrays a failure of understanding of science. Yes, of course, papers are wrong, and there are retractions, and perfectly good papers that aren’t retracted (which is what I meant) will be qualified, modified or even rendered moot by further work.

    But to blame Nature and other journals for the behaviour of ‘pinheaded administrators’ makes no sense. Take it out on them, not us.

  9. @cromercrox
    Yes you are right that it would be wrong to blame Nature for the actions of pinheaded administrators. But I don’t think that Nature can escape all blame, as witnessed by the huge proliferation in Nature titles. Of course Nature is doing only what any efficieant business would do, and maximising its profits.

    That’s to be expected, and the major corruption comes not from Nature, but from academics (or, more commonly, ex-academics) who reward quantity over quality. That is lazy (it’s much easier to count than to read) and does great harm. One advantage of doing away with journals altogether would be to prevent the use of surrogate endpoints (like journal names) and another advantage would be that it would reduce costs enormously as well as increasing public access. It’s an idea that we aren’t quite ready for yet, but I believe it’s well worth thinking about.

  10. cromercrox says:

    ‘One advantage of doing away with journals altogether would be to prevent the use of surrogate endpoints (like journal names) and another advantage would be that it would reduce costs enormously as well as increasing public access’

    I disagree. Costs would rise enormously, as scientists would have to spend a great deal of time filtering the increment from the excrement before they could make any progress.

    Public access might in principle be increased, but would also allow unscrupulous practitioners and credulous journalists to foist on the public a lot of dross without even the most minimal filtering.

    I don’t think the advocates of PPPS really realize how many submissions journals such as Nature receive, let alone other journals. Frankly, to advocate PPPS is to say that everyone should be free to flush their toilets into the street so the public can have the benefit, without those pesky engineers and sewage farms getting in the way.

    I’m not sure I like being described as a ‘surrogate endpoint’.

  11. Mike says:

    Henry, I don’t think Nature (or any other of the few journals with purely professional editorial staff) is a sensible yardstick to judge how costs may or may not change. Most other journals rely on working scientists to do the editorial grunt work, for very small, or zero, financial recompense. Reviewers, who are also working scientists, do not get paid more. If we were to remove scientific publication from the hands of for-profit publication houses, but maintain some system of pre-publication review (and this may, or may not be what David is suggesting), it’s easily conceivable that money could be saved and quality maintained.

    Richard Feynman gave an excellent speech many years ago about scientific integrity, Cargo Cult Science Wikipedia links to pdf and html versions, which all working scientists should be encouraged to read and digest. Some of the problems discussed above arise from scientists not behaving with integrity, e.g., serial submission of a manuscript to ‘lower ranked’ journals without making changes requested in previous reviews that led to rejection. This arises in part due to publish or perish pressures (as David notes).

    A simple way to avoid wasting everybody’s time could be to assign a doi to any MS when it is first submitted (rather than when it’s first published, as happens now), which all subsequent reviews and responses would be permanently attached to. Journal Editors would have easy access to previous reviews and be able to decide more easily if changes from previous suggestions were appropriate, or more reviews required. This has the potential to substantially reduce the editorial and reviewing workload.

    This could prevent unscrupulous behaviour, save time and money, promote scientific integrity and bring your a nice cup of tea in bed each morning.

    And Richard, I agree that PPPR is a poor choice of name for F1000. In part, that’s what scientific publication does anyway. If a paper is worth something (positively or negatively), it will probably get commented on and cited in the literature. There’s your PPPR right there.

  12. Steve Caplan says:

    “So, please, can we drop all this nonsense about doing away with peer review (‘proper’—if you like)?”

    I second the motion.

  13. Of course, there is a cost attached to re-submitting to a journal, even if you DON”T revise the Ms according to any referees’ reports. This is the tedious process of re-casting the paper (inc Figures) into the journal’s particular preferred format. The Glamour End of the journal market (and I’m not just talking about Your Favourite Journals beginning with N, S or C by any means) are particularly nit-picking about this.

    One of my colleagues currently has a paper going through this process. It was laboriously cut trimmed ‘n’ styled for a Fancy-Dan Journal which then rejected it on in-house editorial decision without full review. It has now been re-Trimmed, Cut ‘n’ Styled for a different High-End Journal. If they reject it too my friend will have to do another re-style, and so on and so on.

    Now, I don’t know how much of my mate’s time has gone into all this Trimming, Cutting & Styling, but I dare say quite a bit. Unfortunately, the chance of future funding likely hangs on getting the paper into one of these sort of fancy organs (Wurlitzers?), so s/he feels s/he has no choice.

    Re. the “allocate unique DOI” thing Mike mentions, there are two problems with this that immediately spring to mind. One is that people often DO revise the paper in accord with some of the referees’ suggestions (the more sensible and/or feasible ones). Does that mean a paper should have a new number?

    The other problem is that referee-ing standards do vary at different journals, as do referees. It is true that utter tripe may well find its way out eventually in real bottom-feeding journals, to the general detriment of the literature. But conversely, all working scientists must have had the experience of having a perfectly sound paper rejected for fairly silly but strongly-held views from referees, necessitating the sending of the work somewhere else… and then maybe somewhere else… It doesn’t necessarily imply bad work, just differences of refereeing opinion. And sometimes whether the referee had something for lunch that day that disagreed with them.

  14. @cromercrox

    It’s well known that Nature gets a huge number of submissions, That’s why getting accepted is more like a lottery than a fair process. Again that isn’t Nature’s fault, but it is certainly a fault of the system as a whole.

    You say

    “I disagree. Costs would rise enormously, as scientists would have to spend a great deal of time filtering the increment from the excrement before they could make any progress.”

    That isn’t true. Ever since people stopped going to the library to read paper journals, papers are found by some sort of subject search. I may tirn to Nature for the news, but on the rare occasions when it has a paper of direct interest to me, it comes to my attention in the same way as a paper anywhere else. That’s part of the reason that it has never mattered less where you publish and why many people have papers in low impact journals with more citations than papers in Nature (never forget that there is no detectable correlation between the number of citations that a paper gets and the impact factor of the journal in which it appears). Now,the only reason why it matters where you publsh is to satisfy some bureaucrat who can’t, or won’t, judge quality.

    So we already have to separate the good from the bad, since the bad will always get published somewhere. I must get several emails every week announcing yet another new journal.

    I recall the time when the Journal of Physiology would re-draw Figures for you, and Phil Trans RS would align matrices by hand, but now journals require authors to produce papers in a virtually final form, They do less and less for more and more money.

    Something new has to evolve. I’m ]ust not yet quite sure what it will be.

Comments are closed.