On peer review and trials by Twitter

(Edited to add: just three seconds after hitting ‘publish’ I saw that Girl, Interrupting, has a very fine post making some of the same points. Please read that, too.)

Towards the end of last year, the day job ran an issue on the subject of peer review, very much taking the line that it’s in crisis, that it’s not the best way to do things, yadda-yadda.

That put me in a bit of an uncomfortable position because I didn’t agree at all with the message coming out. It didn’t help matters that we started getting comments along the lines of “You would say that, because you’re now a shill for Faculty of 1000 and they hate peer review too.” But although that is not necessarily true I couldn’t write about it at Naturally Selected, because it would start to look like the Battle of Marston Moor (for my American cousins, think Gettysburg.)

So I kept my head down.

Until Thursday, when Nature published a News Feature looking at peer review and ‘social media’ in the context some high profile controversial papers, Peer review: Trial by Twitter. Various colleagues emailed the story to me (which isn’t in itself all that exciting) because of the criticisms of F1000 that were contained therein. In return I wrote a typically robust response. In brief, I showed how those criticisms were wide of the mark, and I was fuming at the unattributed ‘critics note that F1000 rankings…’ line. (I think I know where this was sourced now, and the stats in the paper are—according to my own source—crap.)

Anyway. There are two real issues here: first, is peer review really in crisis? Second, is the use of ‘social media’ tools damaging to science? (I don’t really care if individual scientists get upset—more on that in a bit.)

As I hinted above, I really don’t think peer review is in crisis (£: letter from Tim Vines et al.), and neither do I agree with Cameron that we should publish everything and let the market decide. The sheer volume of papers makes such a model unworkable (we’re already looking at around 1 million papers yearly in medicine and biology), and even if you could solve the search/filter problem in terms of identifying what you’re looking for, with no barrier to publication how the hell do you flag the crazies, the lazy and the simply inexperienced? Having experienced the full gamut of peer review, from having a manuscript accepted without revision, through having one improved by review, all the way to chasing around for two years to get something published, I’m approaching this without an axe to grind.

Look at it another way: you can already publish your manuscript, open access, with no peer review (or editing, come to that). It’s as easy as typing a blog post. Why aren’t people flocking to do this, if it’s the way forward? Because peer review is the standard supported by the vast majority of practising scientists, and they recognize the value in it. And don’t say to me that the grant funders won’t stand for it: there are plenty of scientists who don’t depend on competitive grants.

I’m not denying there are problems with the current system. The whole anonymity/open review argument is not settled, not by a long chalk. There will be the vindictive and the inexperienced in any human endeavour. Peer review is often deemed—especially by the media and the loons (coughhomeopathscough)—to be a mark of correctness. And we all know it’s not—it’s not even a quality control stamp. Peer review says (or should say) that the work in question, as far as we can tell, has been done correctly, the appropriate controls/stats/ethical approval have been included, and the right literature has been cited. Anything more than that is an editorial decision (‘is this important/general enough for Nature?’ say).

So at F1000 we actually quite like the peer reviewed literature. I do, anyway. With a very few exceptions everything at F1000 has already been peer reviewed—so in a way F1000 does post-publication peer review lite. I started using the phrase ‘post-publication peer review’ in the context of F1000 a couple of years ago, without in any way meaning to imply we should do away with traditional peer review.

By the way, looking at the two examples of high-profile paperswith problems, you’ll notice that they, along with many, many others, are high-profile simply because they have been pimped shamelessly by various parties—the journal, the media, the press offices or even the scientists themselves. I fail to see how strict post-publication peer review will change that.

Moving on, you’ll notice I’m using scare quotes around ‘social media’. That’s because I thinks it’s a bit of shit phrase, to be honest. The internet has been social from the beginning. It started, for goodness’ sake, as bulletin boards and emails. What can be more ‘social’ than that? (As somebody on Facebook said the other day, asking what’s after social media is a bit like asking ‘what’s after porn?’)

Nothing has changed, really. I’ve stood up at big conferences and had to defend my work. I said in my blog post that getting trashed via twitter and whatnot is scary: well, it’s nothing compared with a couple hundred of the best brains in the field trying to pick holes in what you’ve just said. We’ve always, always taken papers apart in private, at conferences, over coffee: now we have the tools to make it happen even faster, and perhaps more importantly give instant feedback, good and bad, to the author.

Which can only be good, surely? Scientists might have to rethink how they respond (and do it faster), but if you don’t want to fight for your work, you shouldn’t be in the job in the first place.

Of course bad things happen on twitter. There’s a lot of noise, but a lot of really good stuff (and for a well-balanced view I commend to you David Kroll’s meta report on Scienceonline 2001) too. It’s what you make it.

And the bottom line is, none of this is going to affect how most of the lay public view science or its process. Most people aren’t on Twitter, and those that are don’t follow active scientists. They are going to get their information, as they always have done, from the media and from celebrities. More to the point, the vast majority of scientists don’t engage with what we are calling ‘social media’ (a highly unscientific straw poll of ~40 Wellcome Trust Fellows revealed one who used Twitter and four that read blogs).

So. This all makes for exciting news articles and opinionated pieces in blogs and whatnot, but frankly, I think it’s so much guff. Peer review will trundle on much as it has done for years, and whatever replaces Twitter will bring prognostications of doom and death and the end of civilization as we know it™, but nothing will change, not essentially, and scientists will continue doing totally awesome things.

It’ll keep editors happy, though.

About rpg

Scientist, poet, gadfly
This entry was posted in F1000, Literature, Rants and tagged , , , , . Bookmark the permalink.

21 Responses to On peer review and trials by Twitter

  1. Shecky R. says:

    A core issue involved here is to what degree, if at all, authors of a published study should respond directly to serious criticisms of their work arising on social media sites… or do they ignore such criticism unless and until it appears in a traditional peer-review print format? Your answer (AND mine) seems to be that “Scientists might have to rethink how they respond (and do it faster), but if you don’t want to fight for your work, you shouldn’t be in the job in the first place”… but a lot of long-term scientists don’t see it that way, or don’t see it as being “professional.”

  2. rpg says:

    Thanks Shecky. I can’t help but point at conference talks, poster sessions even, where authors do respond directly to criticisms.

    I think saying, on SM sites and what not, “I will respond to this, but it’ll be published at XYZ” is fair. At least in the short term. Asking people to answer EVERY tweet or blog post is unreasonable, IMHO.

  3. Benoit says:

    Very nice. I would be pleased, delighted, even ecstatic if someone blogged or tweeted about one of my lab’s papers…someone would have actually read it and cared enough to comment on it! I also would be perfectly happy for constructive criticism, and would mostly be amused by the trolls. Because that’s how science and life works. If our work was over-hyped, or was just plain wrong, that would be a different issue, and then I would deserve it. So I say long live peer-review (it still has its place), and welcome “social media”. Public discourse is what publishing science is all about innit?

  4. rpg says:

    Hah, yes! What Benoit said.

  5. In the more theoretical/mathematical bits of physics there is a resource known as the ArXiv where papers are deposited as preprints. Everyone knows it’s a preprint library but in certain fields (not my own, so I’m no expert, but quantum and high energy physics certainly) it seems to be the main repository. Some people go on to publish in peer reviewed journals, others do not. Yet it is highly valued by the whole community. It would be interesting to know how this fits into the bigger picture because people clearly do feel some of the stuff on it is brilliant and some rubbish and yet they manage to filter through. Doubt the media go trawling through it for big stories though.

  6. rpg says:

    We had a little twitter discussion on Friday about the ArXiv. Bottom line seemed to be that it was really popular in a couple of disciplines, but the (vast?) majority of physicists don’t use it. And even those that do tend to go on to publish ‘traditionally’, as you say.

    I’d like some real data because people hold the ArXiv up as an example of post-pub PR, but I think it’s total BS to think you can extrapolate.

  7. @Athene – if you look at this link – http://scholarlykitchen.sspnet.org/2009/06/17/physics-papers-and-the-arxiv/ – it gives a good description of physics papers which are and aren’t routinely published in the ArXiv journal … Its a good blog post –

    Plus I think ArXiv as you said, and people elsewhere forget to say, perhaps conveniently, is it is a ‘preprint’ archive and certainly most people do publish in a peer-review journal after (or even before) they submit to ArXiv – those papers are still peer-reviewed in the end – for the most part.

  8. Stephen says:

    …it’s not even a quality control stamp…

    Well that’s not quite true (as I think you probably agree). There is some quality control in peer review — otherwise what is to distinguish it from Cameron’s horrendous post-publication peer-review scenario? I guess what you mean is that peer review does not provide the stamp of truth. But what it most certainly does achieve — as you and Girl, Interrupting attest — is to weed out some of the rubbish and to buff up the intellectual or explanatory rigour of the rest.

  9. rpg says:

    Yar…there’s some quality inasmuch as we think it’s science done more or less right, but there’s no guarantee it is right.

  10. Heather says:

    That really makes a nice tandem with Girl,’s post!

    I love the phrase “the crazies, the lazy and the simply inexperienced” – that’s exactly why I am glad of peer review, and always submit my papers to peer-reviewed journals. Otherwise, I could so very easily present the work on the Internet, and hope to accrue criticisms if I could get any attention from people whose opinions might be useful and worth my respect.

    However, that was the point of Open Science at one point, and I was enticed by the message. What if we caught the errors and sloppiness before publication, by sharing the data, time-stamping our hypotheses, conducting group analyses? Well, that is what a good reviewer does. But I have had so many sloppy reviewers. Who watches the watchmen? The editorial board, one would hope, or the editors.

    This is why choosing the journal is such an important decision – the venue does add to the work itself.

    After having submitted some work in Open Access mode (never again; my lab is not rich enough and I’m happy to redistribute PDFs upon request) I get regular requests to submit my articles to “journals” from the Hindawi stable to submit my work to their hitherto unknown title. And they always have the most obscenely long list of people on their editorial boards. I wonder if they actually all agreed to it? I’d get better results from my “peers” by submitting here.

  11. Heather says:

    That last paragraph, for example, could have used some peer review. Short of that, the ability of commenters to modify their own comments once submitted 😉

  12. cromercrox says:

    Excellent. As an editor with your favourite weekly professional science magazine beginning with N, I am firmly in favour of peer review – but then I would say that, wouldn’t I? It’s not that peer-review is perfect, because it’s not. Referees, like authors, and – gasp – editors, are only human. We make mistakes. We publish papers which, on reflection, we probably should have rejected. We reject papers which, on reflection, we should probably have accepted. It is as someone once said about democracy: it’s the worst form of government imaginable, except for all the other ones.

    Peer-review is definitely not in crisis, and here’s why – everyone has a vested interested in it. Everyone wants it to work. The people who referee papers (for free, I might add) are the same people who write papers. It works because it is a do-as-you-would-be-done-by system. It is an honour system, built on trust and mutual respect. Some people think referees should be paid – but that, I feel, would do great violence to that trust.

    I very rarely hear of a referee complaining about having too much to referee, at least in a general sense. However, one regular referee complained to me recently at the increasing load of manuscripts to review as a function of the ever-increasing number of peer-reviewed journals – which rather puts paid to any sense that Cameron Neylon’s stance is in any way popular.

    My role in this, as an editor, is to be ever more discerning about the manuscripts I send out to review in the first place. I will only send out manuscripts to review if I am pretty sure, editorially, that the manuscript deserves a place in your favourite weekly professional science magazine beginning with N, provided that the technical details stack up. I won’t send a manuscript to review if I’m unsure about it, just to see what referees think – that’s an abrogation of my job, and also an abuse of the system, in that I am asking someone (unpaid) else to do my (paid) job.

    Does this make journal editors even more powerful than they already are? Yes. But think of the alternative. Oh, I see Cameron Neylon already has.

  13. Heh. That remark about democracy (which I think was Churchill’s, at least in the widely quoted version) is the same one I use when people ask about the failings of peer review. In fact, I used it only last Monday when I was giving an evening talk to the local Cafe Scientifique about pseudoscience.

  14. Steve Caplan says:

    Everyone- have a look back at the threads for Cath’s LOL-cat blog- there are a couple of replies there by Dr. Neylon to my comments, and a response by me in which I also use the same “Democracy argument”- I do think you are accurate with Churchill (or his speech writer) being given the original credit.


  15. Irene Hames says:

    I was a professional editor for 20 years and, based on my experience of seeing thousands of reviews and thousands of decisions, I, as Henry, am firmly in favour of peer review. I also second many of his comments.

    What always surprises me is that peer review works as well as it does. In journals with professional editors and/or established editorial offices that remain through editor changes, the knowledge and skills needed to run a good peer-review system are there, and new editors can be inducted into the process. But how many of the editors of the world’s 25,000 or so peer-reviewed journals have ever received any training in this? They’re usually appointed on the basis of their research record, reputation, and ‘vision’ for the journal. One day they’re researchers, the next they’re in charge of vehicles that determine the grant prospects and careers of the research community, and how billions of research funding is allocated. I can’t think of any other profession where this happens.

    I also haven’t experienced any real crisis in peer review. But there are problems, some brought on because of what publication has become. It can’t be put better than Stephen Lock did in the introduction to the third impression (1991, pxi) of his book ‘A Difficult Balance: editorial peer review in medicine’ (originally published 1985):

    “And underlying these worries was yet another: that scientific articles have been hijacked away from their primary role of communicating scientific discovery to one of demonstrating academic activity.”

    This has led to work too often being submitted prematurely, sliced too thinly, and some overloaded editors passing on work that is their responsibility to reviewers.

    At its best, with skilful editorial management and a good editor, peer review is a sophisticated process that results in fair, helpful and constructive reviews, good decisions, and significantly improved papers. At its worst, it’s a lottery.

  16. cromercrox says:

    @Irene – you wrote ‘What always surprises me is that peer review works as well as it does’. That amazes me, too.

    Cameron Neylon’s argument seems to be that the publication of a few well-publicized turkeys is indicative of a malaise in the whole system. But for every over-hyped peer-reviewed paper claiming that arsenic-eating bacteria make you live longer and prevent autism, tens of thousands of good, solid papers get published whose particulars are as free from reproach as referees and editors can make them. Were refereeing to be entirely post-hoc, just how many arsenic-eating bacteria would be inflicted on the hapless reader, who’d not have the benefit of any kind of filter, however crude and imperfect?

    You write ‘At its best, with skilful editorial management and a good editor, peer review is a sophisticated process that results in fair, helpful and constructive reviews, good decisions, and significantly improved papers. At its worst, it’s a lottery.’

    To which I shall add – post-hoc peer review will never be anything but a lottery.

  17. Mike says:

    To which I shall add – post-hoc peer review will never be anything but a lottery.

    Is that aimed directly at Richard’s current employers, Henry?

    To avoid repeating many of the excellent points above, I hope that we can accept that peer-review is a vital filter, but that’s no reason to stop asking how we can improve it. Particularly as the publication landscape is continuously changing. Perhaps the peer-review model we still use as the first filter is reaching its sell by date. Shifting the editorial/review process online might not be a sufficient change, it’s almost purely cosmetic. But the question of how to actually improve things remains open and important. As with the rest of science, we should not become dogmatic in our beliefs.

    Answers on a postcard…

  18. Frank says:

    The papers that were heavily criticised were

    high-profile simply because they have been pimped shamelessly

    Live by the sword, die by the sword. Claim high, fall low. The only thing that has changed is the speed with which retribution comes.

  19. Pingback: Science in the Open » Blog Archive » What is it with researchers and peer review? or; Why misquoting Churchill does not an argument make

  20. Pingback: More peer review. Zzzzzz. | Not ranting – honestly