Once again the peer-review vs. science online debate appears!
In an article by Peer review: Trial by Twitter – Apoorva Mandavilli talks about a lot of things but it mentions that science is getting ‘torn apart’ in the online media… which is ‘scary’ @rpg7twit (aka. Richard P Grant) has a nice response to this in the F1000 online magazine Naturally Selected from The Scientist…
OK maybe it is ‘scary’ but, sorry this, as a scientist is part of your damn job, presenting your work. It isn’t always pleasant going through peer-review, but it is a part of the process of presenting your science. And if your science makes it into a spot in one of the ‘big’ journals – like Nature well you want people to read it, or you wouldn’t have put it there in the first place. And like movie stars, you can’t have the fame without the Paparrazzi – that is you can’t only have praise with no critique, sorry life doesn’t work that way. Science is about, for better or worse, proving your hypothesis, it has to stand up to the test of time. It occurs to me perhaps the ‘test of time’ is just accelerated by the social media/internet process. This is a GOOD thing – maybe not for the scientist who got something wrong, but wake up – scientists get things wrong.
And it seems to me, as has been said, all this ‘taking apart of papers online’ just shows the public what scientists actually do – they take apart – or agree with – other’s work and this leads them to retest or look for new phenomena – that is a part of what we are supposed to be doing as scientists.
A few years ago in a field related to my own – somebody published a finding which said that liquid water (as opposed to being a tetrahedral network as most people have measured that it is) was not tetrahedral but actually linear. This, most people in the field thought, was crap. But instead of them all just sitting around and saying online it was crap – which they did – they also went back and re-measured and retested old data on liquid water. And as a much respected colleague of mine said at the time – this probably wasn’t such a bad thing for our field it shook it up and got people really explaining in better detail than before WHY the data showed what it WAS.
Tetrahedral water network in ice
But this isn’t actually why I am writing this blog post.
One thing that seems to come up in this debate often is something that is advocated by Cameron Neylon quite strongly which is:
‘it makes much more sense to publish everything and filter after the fact’
And this is where I’d like to archly raise an eyebrow myself.
I really cannot see, nor have I heard a very good argument as to why this makes more sense?
Its hard enough trying to filter through all of the work IN peer review journals if people just publish everything, imagine the volume. And imagine the volume of crap – as I pointed out in a blog post about a year ago – the internet now, and just the newspapers in the past already sometimes serve this function – look at the cold fusion story – where the science never even made it to peer review.
Also what if a ton of people think its crap but it really isn’t crap – which would be sort of opposite of the recent arsenic story.
This sort of implies and it has been implied that peer-review as it stands now is a bad thing
Why do so many people think everything is wrong with peer review? Perhaps it is these articles we see on Twitter – the ones where we see where the peer review process has missed something, but there are far more stories about where this doesn’t happen, I would suspect, you just never hear about them.
Does peer review go wrong? Yes
Do we hear about where peer review failed in the news? Yes – think of the recent example of arsenic.
Are there problems with the peer-review process? Of course.. look at Jan-Hendrick Shoon and recently Amil Potti
Do most peer-reviews articles NEVER make it to the news? Yes
But will publish first filter later make this better? I really doubt it.
Peer review is a team of people who are ‘experts’ in your field who look at your work and assess it – why is this altogether a bad thing?
In fact it can be, and largely is, a good thing. I have written papers – especially my first papers as corresponding author, where I was so focused on the forest I didn’t just miss the trees I actually, unknowingly, cut down the trees. After these manuscripts were peer reviewed – it was clear to me, from the reviewer’s comments – that they had no idea what I was talking about. Why? Because I wasn’t at all clear – so I had to fix it and as a result had a better paper. I have also had reviewers suggest work to me by other that I wasn’t aware of, often in agreement of what I was saying. That not only strengthened what I was saying in a particular paper but also my research in general.
If everything was published first – would be people take the time to even look at my piece of research to tear it apart? I am not an FRS or a Nobel prize winner, nor am I likely to be, but when I submit my paper to a suitable journal I have the same equality as anyone else in getting it read (this isn’t true for all journals, but specialists journals, certainly) – and I want it to be ‘torn apart’ to see if my science withstands the test because THIS is the point, whether it scares me or not.
Here, here. You are absolutely correct with your assessment of the peer-review process. I read that comment by Cameron Neylon about letting the readers “filter out” the bad papers, and it sent shudders down my spine. In today’s highly specialized world of science, there is absolutely no way that even informed readers (and we are frequently reading into peripheral and tangential fields) are going to be able to decide what is scientifically accurate and what isn’t.
Reading a paper for the purpose of planning future experiments or general knowledge is extremely different from the exacting peer-review to which papers are subjected. As I commented in an earlier response to this same issue, doing away with peer-review in science (with all its inherent flaws) would set scientific research back aty least 50 years.
Very glad to hear that other scientists agree with this prognosis…
Let me latch on to that Cameron Neylon comment also, and call it rubbish. I, for one, have far better things to spend my time on than reading ten times as many papers as I already do, and sort out which nine are garbage. That is what we (individuals, institutions, or purveyors of advertisements on open journal websites) are paying for – the editorial and peer-review infrastructure and process.
Ricadipus-
Paying for peer review? Where do I collect my check?!
Pay reviewers? Um, no. If so, sign me up too. But not too often…
What I meant, of course, was the journal subscriptions, at least a small portion of which go to pay for the overheads involved in administering the peer review process – even with online submission, it still takes time and some portion of the cost of IT resources for the journal.
Drat–another penny lost…
Ooh I wish – thanks for your comments – on a side note I do think double blind peer-review might be a good idea too – no author names – but though its messy I think it works rather well
I’ve tried double-blind peer review. It doesn’t work. You can always work out who it is.
I’m so embarrassed. I wrote this, and then saw your post. I had no idea.
RPG–you were just scooped…
don’t be – great minds think alike – and fools never differ! I liked your post
I’d written most of the post, then went out to see The King’s Speech (superb movie, by the way): came back, cooked dinner, edited and uploaded… then saw this. Very funny.
I think its great, personally… all brought to you by Occam’s T – and just shows that ideas are not, after all, very unique – at least when it comes to ‘what we should do about peer-review’
thanks for linking me by the way…
Double-blind peer review is nice in theory, but in my field it would be practically impossible NOT to know whose paper is being reviewed. But the reviewers not knowing who else is reviewing is a major motivating factor for good review–not looking silly (too harsh or too lenient) in front of one’s peers.
that is the interesting thing about peer-review – I had a boss who was one of the editors of Physial Review Letters and he told me that more than 1/2 of the time when authors were annoyed about a reviewer – saying they ‘knew’ who it was and that person was biased against them, etc – the authors had often gotten it wrong and the reviewer was one of the people actually recommended by the authors themselves. The point is – I think even in small fields you might think you know, but with double-blind you are never quite sure … and this might allow more accuracy. In economics for instance it resulted in far more people being about to ‘break into’ a field – eg more junior faculty got published as a result of double-blind reviewing – there is a link to the study in my old blog on this (linked above) – anyway – intereting thoughts…
I agree–but only in very specific cases. So many of our studies are follow ups on things we have previously done, so it would be obvious even if we were to remove the “we have previously shown” and replace it with “it was previously demonstrated”…
yes that is true …. I say that too – so then they know…
Good conversation. As in so many spheres of life, it’s very good to have idealists around to help us all define what we really want and at what cost. I’m always glad when Cameron shakes it up a little. And I wanted to just chip in to say that there are additional readers out there, lurking around, and I was one until just now 🙂
Just to say I agree. When I get peer reviewers comments I always think they are written by a foolish ignoramus, but I usually come round to realising they are worthwhile. Found myself defending peer review against an attack by Richard Smith recently: http://tinyurl.com/2etogtt
It’s painful and it can go wrong, but it’s a system that works on balance.
Yes I read that article you wrote (and I think, I hope) retwteeted it – I thought it was excellent…
Excellent post. Couldn’t agree more. bravo.