(Edited to add: just three seconds after hitting ‘publish’ I saw that Girl, Interrupting, has a very fine post making some of the same points. Please read that, too.)
Towards the end of last year, the day job ran an issue on the subject of peer review, very much taking the line that it’s in crisis, that it’s not the best way to do things, yadda-yadda.
That put me in a bit of an uncomfortable position because I didn’t agree at all with the message coming out. It didn’t help matters that we started getting comments along the lines of “You would say that, because you’re now a shill for Faculty of 1000 and they hate peer review too.” But although that is not necessarily true I couldn’t write about it at Naturally Selected, because it would start to look like the Battle of Marston Moor (for my American cousins, think Gettysburg.)
So I kept my head down.
Until Thursday, when Nature published a News Feature looking at peer review and ‘social media’ in the context some high profile controversial papers, Peer review: Trial by Twitter. Various colleagues emailed the story to me (which isn’t in itself all that exciting) because of the criticisms of F1000 that were contained therein. In return I wrote a typically robust response. In brief, I showed how those criticisms were wide of the mark, and I was fuming at the unattributed ‘critics note that F1000 rankings…’ line. (I think I know where this was sourced now, and the stats in the paper are—according to my own source—crap.)
Anyway. There are two real issues here: first, is peer review really in crisis? Second, is the use of ‘social media’ tools damaging to science? (I don’t really care if individual scientists get upset—more on that in a bit.)
As I hinted above, I really don’t think peer review is in crisis (£: letter from Tim Vines et al.), and neither do I agree with Cameron that we should publish everything and let the market decide. The sheer volume of papers makes such a model unworkable (we’re already looking at around 1 million papers yearly in medicine and biology), and even if you could solve the search/filter problem in terms of identifying what you’re looking for, with no barrier to publication how the hell do you flag the crazies, the lazy and the simply inexperienced? Having experienced the full gamut of peer review, from having a manuscript accepted without revision, through having one improved by review, all the way to chasing around for two years to get something published, I’m approaching this without an axe to grind.
Look at it another way: you can already publish your manuscript, open access, with no peer review (or editing, come to that). It’s as easy as typing a blog post. Why aren’t people flocking to do this, if it’s the way forward? Because peer review is the standard supported by the vast majority of practising scientists, and they recognize the value in it. And don’t say to me that the grant funders won’t stand for it: there are plenty of scientists who don’t depend on competitive grants.
I’m not denying there are problems with the current system. The whole anonymity/open review argument is not settled, not by a long chalk. There will be the vindictive and the inexperienced in any human endeavour. Peer review is often deemed—especially by the media and the loons (coughhomeopathscough)—to be a mark of correctness. And we all know it’s not—it’s not even a quality control stamp. Peer review says (or should say) that the work in question, as far as we can tell, has been done correctly, the appropriate controls/stats/ethical approval have been included, and the right literature has been cited. Anything more than that is an editorial decision (‘is this important/general enough for Nature?’ say).
So at F1000 we actually quite like the peer reviewed literature. I do, anyway. With a very few exceptions everything at F1000 has already been peer reviewed—so in a way F1000 does post-publication peer review lite. I started using the phrase ‘post-publication peer review’ in the context of F1000 a couple of years ago, without in any way meaning to imply we should do away with traditional peer review.
By the way, looking at the two examples of high-profile paperswith problems, you’ll notice that they, along with many, many others, are high-profile simply because they have been pimped shamelessly by various parties—the journal, the media, the press offices or even the scientists themselves. I fail to see how strict post-publication peer review will change that.
Moving on, you’ll notice I’m using scare quotes around ‘social media’. That’s because I thinks it’s a bit of shit phrase, to be honest. The internet has been social from the beginning. It started, for goodness’ sake, as bulletin boards and emails. What can be more ‘social’ than that? (As somebody on Facebook said the other day, asking what’s after social media is a bit like asking ‘what’s after porn?’)
Nothing has changed, really. I’ve stood up at big conferences and had to defend my work. I said in my blog post that getting trashed via twitter and whatnot is scary: well, it’s nothing compared with a couple hundred of the best brains in the field trying to pick holes in what you’ve just said. We’ve always, always taken papers apart in private, at conferences, over coffee: now we have the tools to make it happen even faster, and perhaps more importantly give instant feedback, good and bad, to the author.
Which can only be good, surely? Scientists might have to rethink how they respond (and do it faster), but if you don’t want to fight for your work, you shouldn’t be in the job in the first place.
Of course bad things happen on twitter. There’s a lot of noise, but a lot of really good stuff (and for a well-balanced view I commend to you David Kroll’s meta report on Scienceonline 2001) too. It’s what you make it.
And the bottom line is, none of this is going to affect how most of the lay public view science or its process. Most people aren’t on Twitter, and those that are don’t follow active scientists. They are going to get their information, as they always have done, from the media and from celebrities. More to the point, the vast majority of scientists don’t engage with what we are calling ‘social media’ (a highly unscientific straw poll of ~40 Wellcome Trust Fellows revealed one who used Twitter and four that read blogs).
So. This all makes for exciting news articles and opinionated pieces in blogs and whatnot, but frankly, I think it’s so much guff. Peer review will trundle on much as it has done for years, and whatever replaces Twitter will bring prognostications of doom and death and the end of civilization as we know it™, but nothing will change, not essentially, and scientists will continue doing totally awesome things.
It’ll keep editors happy, though.
