If this paper is the future of open access publishing, then we are in for an interesting ride. And it’s a journey that will reveal a great deal more about the process of science than most outsiders will have seen hitherto.
The article by Kinch and Grishin, published in a recent issue of the very open access journal Biology Direct, provides a great example of the self-correcting nature of science by refuting a finding from Kiriakidou et al. that was reported two years earlier in one of the life science giants, Cell. What adds a particular frisson of excitement to all this is that the reviewers’ comments and the authors’ responses are appended to the end of the Biology Direct paper for all to see. I’ve not come across this level of openness before, but I think it could be a very good thing.
The scientific nub of the matter is of considerable interest since it touches on an exciting topic—translational repression by RNA interference (RNAi)—but the public nature of the dispute is even more gripping.
Kiriakidou and colleagues looked at the amino acid sequence of human Argonaute, a protein crucial to the mechanism of gene silencing by RNA interference. Curiously, they found a segment of Argonaute that resembles another protein, eIF4E, the mRNA cap-binding protein that controls the initiation of protein synthesis. The inference from this observation, that Argonaute might also bind the mRNA cap, appeared to be borne out by experiment and led to the formulation of a plausible mechanism to explain how the synthesis of proteins from mRNAs targeted by Argonaute might be repressed. Here was an important new finding in a hot area of science. No surprise then that it ended up being reported in Cell. The paper was even flagged up by the Faculty of 1000.
But there’s a problem: the Kiriakidou paper is based on an error. Kinch and Grishin looked again at the protein sequences of Argonaute and eIF4E—using more sophisticated methods—and detected no similarity. None.
More compellingly still, they compared their three-dimensional structures of the two proteins (which, curiously, were both available in 2007) and showed that they were not the same. At all. As a result, the title of their paper in Biology Direct reads like an inversion of the earlier work: Argonaute ‘does not contain an eIF4E-like mRNA cap binding motif’. We are a long way here from the linear progression model of scientific development, but somewhat closer to reality.
And what brings us closer still is the publication alongside the paper of the correspondence between the reviewers and the authors, which gives some great insights into the messy and disputatious business of science.
In their comments the authors reveal that the paper was originally submitted to “the high-profile journal that published the Kiriakidou et al paper” (tellingly, Kinch and Grishin can’t even bring themselves to write the name of the journal) but was rejected after a six-month wait “on the basis of lacking experimental evidence”, even though the reviewers for Biology Direct are unanimous in their praise for the thorough-going nature of the work. It does rather look as if Cell has dodged a dose of medicine that might have done the journal and their readers (and authors?) some good.
While the referees are generally very positive in their comments, there is a request to shorten some of the sections describing the methodology of the paper, but Kinch and Grishin defend the inclusion of such details as necessary to convince their peers of the utility of computational analyses of sequence and structure. Again a certain frustration is evident: “Without a certain detailed familiarity with the methods, sequence similarity search results are easy to misinterpret” and go on to complain that many life scientists are simply too ready to discount “computational experiments”. Could they be referring to prejudice? My favourite remark is their parting shot: “Although convincing researchers with pre-formed opinions is particularly difficult, we think our description contains an educational component that could help developing minds.” Ouch.
Not every paper is so controversial but this sort of episode is common enough in science. It’s a business that is messy, incomplete, contradictory and suffused with human feeling. Often things have to be thrashed out in several labs before a consistent and plausible story emerges. Not every experiment is perfectly insightful or free from error. And day-to-day battles may occasionally be won on the strength of feelings about this or that piece of data. But the process is nevertheless a sound one; in the long run, through repeated experiments and thorough discussion, the data will speak and well-founded skepticism will suck out the traces of personal venom. It’s still the best way we have to fight the war of discovery.
And it’s good that this messiness should be made known, so that the public get a much better understanding of what science is really like. This seems to be a growing movement; just recently we have seen Alom Shaha’s brilliant film on the importance of science and, as Ryan and Heather have pointed out, a great new web-site at Berkeley that aims to present the warts’n’all view of what scientists do. The movie mentioned recently by Caryn looks to be in the same vein.
The publication of the reviewers’ comments also strikes me as a good way to make sure that they do a professional job of their critique. Biology Direct even goes one step further in publishing the names of the reviewers (all members of the Editorial Board). As a sometime reviewer myself I would be happy to live with publication of my comments on someone else’s manuscript but I’m not sure I’d want to have my name printed alongside. I guess I would be apprehensive about repercussions on my own manuscripts and grant applications from aggrieved authors, concerns that are shared by others.
Kiriakidou and colleagues will no doubt have been stung by the correction — no-one likes to be proved wrong (I know I don’t). But they’ll get over it. I’m sure they know they made a mistake and will be determined not to repeat it. In any case their experimental observation that Argonaute can bind mRNA caps still stands and may be significant, though it is likely to come under renewed scrutiny. But this is good; it will generate new data and new insights into how the protein works.
That’s the funny, amazing thing about science: it always wins.
Kinch, L., & Grishin, N. (2009). The human Ago2 MC region does not contain an eIF4E-like mRNA cap binding motif Biology Direct, 4 (1) DOI: 10.1186/1745-6150-4-2
Interesting stuff. Thanks for sharing that. As a PhD student about to leave grad school I find the publishing process both extremely difficult and frustrating, but yet so rewarding. I am also lucky enough to help referee papers, which is an eye opening and educational experience. I, like you, am a little unsure if I would want my name along with my critique out in the open. Keep up the interesting posts.
Cheers Matthew – thanks for the comment. It sounds like you’re getting some good experience in your PhD. I didn’t get into reviewing until later though, now that you mention it, maybe I should be involving my own students a bit more.
if you are interested in more about Biology Direct , we (_Nature_) published an article by its founding editors as part of our peer-review debate in 2006. It is archived on the peer to peer blog here . I think it is a very interesting article and an interesting model of publication. I like the idea of posting peer-reviewers’ comments, but it is not entirely straightforward, for the reasons you point out and others – perhaps some of them more specific to journals where there is a lot of editorial input/judgement and several rounds of major peer review before the paper is published.
One little experiment we tried a year or two ago and which I really like is the “Inside the paper” feature, where the editor creates a narrative to accompany the editors’ and author’s initial assessment, the peer-reviewers’ reports (which in this case could be signed or not, depending on the reviewer’s preference), author’s response, rationale of editor’s decision, and subsequent iterations before publication.
Thanks for the link Maxine – it provided an great exposition of their novel reviewing model. I see I’m only 3 years behind the zeitgeist – not bad for me!
I agree it’s not a model that could so readily be applied in all other journals and situations. In many cases the correspondence back and forth would be far longer than the manuscript itself (though sometimes far more interesting too!). A summary format might work but who would want to take the trouble to write that? Alternatively the reviews could be structured so that the major criticisms were discussed first, followed by minor remarks, with only the comments on the substantive issues then being published.
But I think more openness, however implemented, has to be the way to go.
who would want to take the trouble to write that?
Yes, that’s the fatal flaw in the plan, I think 😉
An enjoyably readable case-study of the process, which highlights what many in the media and those anti-science types in some humanities’ faculties often fail to inform of, or acknowledge: that science is conducted by humans; and whence occurs error, or omission, or misinterpretation – or fraud – it is science (hence, human scientists) that sorts the stink out. It isn’t perfect, of course; in the meantime, careers can be damaged, and the ‘correct version’ can be delayed, wasting the time of researchers who might be led down wrong paths. But again, that’s part of the process. Publishing reviewer’s comments certainly merits consideration, because it might make some of them read the damn thing properly.
Agreed Maxine – I’m all for reducing the academic (and editorial) workload. But we must be smart enough, surely, to figure out a way to do this. BTW – sorry you didn’t make it to the excellent London pub night last Wed!
Lee, I guess there is a risk that if we wash too much of our dirty linen in public, the public may be more wary of the value of scientific outputs. But I think it’s a risk worth taking. As you say, the openness will have the benefit of improving the quality of peer-review.
FYI, there has been some further discussion of this post over at Friendfeed. Looks like publication of reviewers’ comments has also been tried out at PLoS ONE.
Yes, PLOS One does post reviewers’ comments. There is a very good recent post on Nascent by Euan Adie in which he analysed online comments at BMC journals (not sure without checking if Biology Direct was included) and PLOS One, which included the posting of refs’ reports (sometimes by journal staff). There were a few posts in the series, but the most recent one is here and you can go back from there.
The BMJ also have this option, though they never published reviewers’ names. Wikipedia has a summary of other open peer review initiatives.
Thanks for the links Maxine and Frank – both v. interesting. I see Nature decided not to pursue open peer review following their trial – but has there been any thinking about other avenues?
I’m glad I read this, I just picked up the Kiriakidou paper over the weekend, thinking ohmygawd, they’re close to unlocking the mechanism behind translational inhibition in animals. turns out it was close but no banana. ah well. i find that a lot though, I read one paper with marvelous results only to read another that was published afterwards with contradictory results. does feel like we’re in the trenches, a bit. it’s interesting to find out what goes on behind the scenes of a publication with the peer reviews.
If you’re interested in RNAi Linda — and like a good ol’ fist-fight — you might enjoy the combative style of Marilyn Kozak. Just the title of her recent review in Gene will give a flavour of the contents:
Faulty old ideas about translational regulation paved the way for current confusion about how microRNAs function.
You may want to check out the primary literature for yourself though.
Thanks for the link Stephen! Kozak really doesn’t hold back on the critiquing, it’s kind of refreshing. but wow, does she have guts!
Well – this is an interesting development from the EMBO Journal (part of the Nature Publishing Group) – they are going to start publishing reviewers’ comments and authors’ responses as part of a new transparent editorial process.
Hear! Hear! (Shame about the opt-out clause though).
Ah, so I finally read this post. Interesting story that I’d heard about somewhere else, but your summary lays it out nicely. And well done on getting this into OpenLab 2009!
Well, well, well, would you look who it is… finally!
Glad you enjoyed it Richard – and thanks!