Once again the peer-review vs. science online debate appears!
In an article by Peer review: Trial by Twitter – Apoorva Mandavilli talks about a lot of things but it mentions that science is getting ‘torn apart’ in the online media… which is ‘scary’ @rpg7twit (aka. Richard P Grant) has a nice response to this in the F1000 online magazine Naturally Selected from The Scientist…
OK maybe it is ‘scary’ but, sorry this, as a scientist is part of your damn job, presenting your work. It isn’t always pleasant going through peer-review, but it is a part of the process of presenting your science. And if your science makes it into a spot in one of the ‘big’ journals – like Nature well you want people to read it, or you wouldn’t have put it there in the first place. And like movie stars, you can’t have the fame without the Paparrazzi – that is you can’t only have praise with no critique, sorry life doesn’t work that way. Science is about, for better or worse, proving your hypothesis, it has to stand up to the test of time. It occurs to me perhaps the ‘test of time’ is just accelerated by the social media/internet process. This is a GOOD thing – maybe not for the scientist who got something wrong, but wake up – scientists get things wrong.
And it seems to me, as has been said, all this ‘taking apart of papers online’ just shows the public what scientists actually do – they take apart – or agree with – other’s work and this leads them to retest or look for new phenomena – that is a part of what we are supposed to be doing as scientists.
A few years ago in a field related to my own – somebody published a finding which said that liquid water (as opposed to being a tetrahedral network as most people have measured that it is) was not tetrahedral but actually linear. This, most people in the field thought, was crap. But instead of them all just sitting around and saying online it was crap – which they did – they also went back and re-measured and retested old data on liquid water. And as a much respected colleague of mine said at the time – this probably wasn’t such a bad thing for our field it shook it up and got people really explaining in better detail than before WHY the data showed what it WAS.
But this isn’t actually why I am writing this blog post.
One thing that seems to come up in this debate often is something that is advocated by Cameron Neylon quite strongly which is:
‘it makes much more sense to publish everything and filter after the fact’
And this is where I’d like to archly raise an eyebrow myself.
I really cannot see, nor have I heard a very good argument as to why this makes more sense?
Its hard enough trying to filter through all of the work IN peer review journals if people just publish everything, imagine the volume. And imagine the volume of crap – as I pointed out in a blog post about a year ago – the internet now, and just the newspapers in the past already sometimes serve this function – look at the cold fusion story – where the science never even made it to peer review.
Also what if a ton of people think its crap but it really isn’t crap – which would be sort of opposite of the recent arsenic story.
This sort of implies and it has been implied that peer-review as it stands now is a bad thing
Why do so many people think everything is wrong with peer review? Perhaps it is these articles we see on Twitter – the ones where we see where the peer review process has missed something, but there are far more stories about where this doesn’t happen, I would suspect, you just never hear about them.
Does peer review go wrong? Yes
Do we hear about where peer review failed in the news? Yes – think of the recent example of arsenic.
Are there problems with the peer-review process? Of course.. look at Jan-Hendrick Shoon and recently Amil Potti
Do most peer-reviews articles NEVER make it to the news? Yes
But will publish first filter later make this better? I really doubt it.
Peer review is a team of people who are ‘experts’ in your field who look at your work and assess it – why is this altogether a bad thing?
In fact it can be, and largely is, a good thing. I have written papers – especially my first papers as corresponding author, where I was so focused on the forest I didn’t just miss the trees I actually, unknowingly, cut down the trees. After these manuscripts were peer reviewed – it was clear to me, from the reviewer’s comments – that they had no idea what I was talking about. Why? Because I wasn’t at all clear – so I had to fix it and as a result had a better paper. I have also had reviewers suggest work to me by other that I wasn’t aware of, often in agreement of what I was saying. That not only strengthened what I was saying in a particular paper but also my research in general.
If everything was published first – would be people take the time to even look at my piece of research to tear it apart? I am not an FRS or a Nobel prize winner, nor am I likely to be, but when I submit my paper to a suitable journal I have the same equality as anyone else in getting it read (this isn’t true for all journals, but specialists journals, certainly) – and I want it to be ‘torn apart’ to see if my science withstands the test because THIS is the point, whether it scares me or not.
(1) Peer review as we know it is a recent innovation. Much of the foundation of science was done without it.
(2) Filter after the fact works well for blogs.
(3) If nobody can be bothered to proof-read our work, maybe the problem is with the work itself.
a) science was only ‘professionalized’ in the last centure or so – part of the reason it is recent
b) blogs aren’t science reporting
c) really? there is lots of good work out there that gets missed – depending on the journal it is in, etc… etc… the volume is so large its hard to keep up… unless you are in a tiny field where you know everyone.
PLoS One essentially publishes anything that is correct without regard for significance. The filtering is done after publication. It is one of the fastest rising journal in the history of journals. And some of the most cited papers in the last five years appeared there.
I think there is a confusion over Open Source vs. Peer review perhaps? just a thought – thanks for your comments
Open Access is a related but distinct issue.
Both Open Access and Publish-and-filter are enabled by the Internet where publication is essentially free.
PLoS One has used both of these paradigms…
You wrote: “look at the cold fusion story – where the science never even made it to peer review.”
I have a collection of 1,200 mainstream peer-reviewed journal papers on cold fusion, copied from the library at Los Alamos. I suggest you review this literature before commenting on this research. See:
http://lenr-canr.org
Mainstream?
why are there no PRLs? or Nature papers?
some of these aren’t even related to cold fusion or are published only in ‘cold fusion’ journals –
Mainstream physics journals, um no
The Journal of Electroanalytical Chemistry where the first results were published is certainly mainstream. The topic itself never became mainstream because of the apparent lack of reproducibility. That is, the problems were identified after the work appeared in a reputable journal. Cold fusion is a case where the real peer review happened after publication.
It was released in the news before it was published anywhere – (see the Book Voodoo Science by Bob Parks) – so in that respect it was filtered before it reached the peer-review literature. Yes some of it is published in small journals now, or indeed from Journals set up by people who already believe in cold fusion – which isn’t the same as a main-stream peer reviewed journal.
I see you have published in Chemical Physics according to your web page. It is by the same publisher, and it has roughly the same impact factor as the Journal of Electroanalytical Chemistry. I am sorry, but I am not willing to dismiss you as a crank on this basis. You wouldn’t be such a good blogger if you thought that publishing was all about venues. I don’t think you are a snub.
BTW I would have problems getting people to look at my work without the current system. I am no different. But I come to a different conclusion: I think there is something wrong with my work when I cannot get strangers to comment on it without a formal process. We may need to rethink how we report science.
I am getting confused are you arguing that cold fusion was reasonably presented ?
and I think perhaps I am not being clear….
That JEC paper doesn’t talk about cold fusion per se- it talks about ‘excess heat production’ which is fine and it was peer-reviewed – and it didn’t make some huge claims about cold fusion
Its what happened AFTER that which was nuts – the claims of cold fusion etc – the press-release before peer-review publication problems -perhaps just an overinterrpretation of results – or who knows what – that is the point – there were huge claims made about the results BEFORE anyone had a chance to peer review it – with something that important, or if I had a result that important I would sure try to get it into a big journal. Most science published is more incremental not huge leaps forward – cold fusion would be a huge leap forward.
I think you are correct, science needs to be reported in different and maybe more ways – but I think peer-review is very important and wouldn’t want to throw it on the scrap heap
Peer review is extremely important. We agree.
But I also think it is extremely difficult to determine what is going to be important in the early stages. It takes year, even for the authors, to determine how important their work is. At best you can hope, maybe, to determine whether the work is presented correctly.
I think we agree on both counts
Yes it does take more than just a year for what is important – many things that are of no importance at the time become hugely important years later (lots of examples)
But this is my problem with the whole cold fusion thing – they leapt to huge conclusions prematurely
(I meant “years” though I wrote”year”.)
Back to the original point, I think that you hint that Cameron is opposed to peer review… but he is not! Neither am I.
I will go further: science cannot exist without peer review. It is not just a nice thing… it is what defines science. I can go, pick a paper of yours and decide I don’t trust you. This is allowed in science. I could challenge you no matter how wealthy or smart you are.
But the conventional process we use for peer review was meant for a pre-Internet world. We will grow out of it.
I agree with that!
Yes, I think Cameron (from what he said to me on Twitter) was misquoted perhaps in the article I quote…. he says he will write a blog post about it – I am looking forward to it
Thanks for your comments and the discussion
I would also like to add that cold fusion is considered a pathological science by many such as the US Department of Energy – who found the evidence ‘not persuasive’
see http://en.wikipedia.org/wiki/Cold_fusion for a description
Bravo! I really enjoyed your perspective. Of course,I often wonder about the exclusive reviewers echo chamber clubs and wonder if some areas of discovery would be regarded as highly if research was open to public scrutiny before publishing.
Nice take on the contradiction of notoriety without critique.
It does make me wonder how presentation of research will adapt or make use of newer and pervasive forms of communication and media.
Interesting Horizon on BBC in the UK tonight in a related area:
Science Under Attack http://www.bbc.co.uk/programmes/b00y4yql
Large sections of the public are often not equipped or prepared to understand the real benefits of peer review processes at all, however the information is published.
Such disagreement and debate is presented by some vested interests as a reason to consider science to be in a mess or floundering rather than as a healthy way to progress.
It appears that the general public and media view of ‘Professionals’ is that they are expected to know all the answers and be correct all the time. Politicians, doctors and scientists alike are held to be absolute, unchanging and conclusive in their views and understanding.
Somehow in an age where there is so much information published on the Internet by so many varied sources and with such varied quality, science has to re-educate the masses on how real facts are uncovered, how understanding is built from debate and how being wrong can be a big part of getting it right.