I should have done this a long time ago but I was too proud. I think I need to review.
In 2002 Dan Carter published a paper in BBRC describing the crystal structure of the protein, human serum albumin, complexed with hemin. It was a very interesting piece of work, not least because it was the first determination of an albumin structure from crystals that had been cryo-cooled all the way down to -173°C. This was an important advance because the cryo-cooling reduces the awful damage that X-rays mete out to proteins as they rip through the crystals in the experiments used to determine the protein structure. Basically, if you can freeze your crystals, you can get better data.
But the paper had one glaring omission: the methods used to cool the crystals were not described.
I was intrigued. No, truth be told, I was annoyed. Because we were investigating the same protein and, try as we might, we hadn’t got cryo-cooling to work. Instead, we sealed our precious crystals in thin-walled glass capillary tubes at room temperature, using a technique first developed by the ingenious and irrepressably bohemian JD Bernal in the 1930’s. The capillary method works well as long as your crystals are big enough (fortunately, ours usually were) but tends to limit the resolution of the X-ray data. It was largely superceded by cryo-cooling in the mid-1990’s, but the cooling can be tricky to get to work in some cases.
Chill, baby. Cryo-cooled crystal in a nylon loop
I wrote to the editor to complain about the omission from Carter’s paper and got short shrift. BBRC was a rapid publication journal, he shrugged, and the referees must have missed it. I should take the matter up with Carter.
But I didn’t. I hesitated. And that hesitation turned into a pause that lasted years. We had tried to get the cryo-cooling method to work ourselves but without success. Albumin is a notoriously flexible protein and the crystals can be very fragile. They weren’t easy to grow in the first place and we were reluctant to keep ruining them in failed cooling attempts; the students needed to get their PhDs after all, and the postdocs their papers. Clearly we should have tried harder but we surmised that the crystals were just too delicate to withstand the shock of rapid cooling. I didn’t want to admit our lack of progress to Carter, who was our main competitor in the field, so we soldiered on with our capillaries.
Even so, we didn’t do too badly. In fact we’ve done rather well (as I’ve mentioned before) and published about 20 papers on the structure of albumin and the various molecules that bind to the protein on its circuitous journeys through the bloodstream. It’s been a very rich seam but we’ve mined heavily and are now moving on.
Other labs are now getting in on the act and I would like to think that–in part–they have been stimulated by our work. So when I was asked recently to put together a review article on the crystallographic analysis of albumin, I decided to add some further impetus by including a lengthy section summarising in detail the techniques that we and others have used to grow albumin crystals and to handle them in X-ray diffraction experiments. This brought me back to the vexed question of cryo-cooling. More recently, a couple of new groups have started to apply this method to albumin, but not all have published details. A lab in China had reported a structure determined from cryo-cooled crystals of albumin but, like Carter, omitted their methods.
So back in February I emailed the Chinese group. And because it was only fair, I gulped down my pride and finally contacted Carter to ask about his 2002 paper.
The Chinese group leader replied quickly and candidly. They had been reluctant to include the method because it was unreliable. That was no great surprise to me, given what I knew about the delicacy of albumin crystals. But, I argued, that very unreliability was a reason to publish, not to withhold the methodology. Reporting it would give someone else the chance to grab the baton and run with it. We discussed back and forth in a few cordial email exchanges. I still don’t know what their methods are, but they have at least agreed to report them in detail in their next paper. I’ll be sure to watch out for it.
From Carter, I’ve heard nothing. I waited a couple of weeks and sent a polite reminder. Still nothing. My guess is that he won’t reply and than his silence may be due to his commercial interests in the protein (though I don’t know for sure – just in case he’s litigious!). I’ve no problem with people wanting to develop their results into valuable new technologies, but if they want to publish papers and play the science game, then I want them to play straight. No funny business.
So the next time you are reviewing a paper and forcing yourself to re-read the lifeless prose for the umpteenth time, please don’t be tempted to fast-forward through the Materials and Methods. Give them all the attention they deserve. And then some. And then some more.
Now, where did I leave that piece of humble pie?
Stephen, that’s fascinating, and I am disappointed that the editorial team of both journals let the side down so severely. But I’m not sure you realize that you have a wonderful ace up your sleeve now — your should mention, candidly, in your review how purposefully omitting methods has held back the field. A review often is an appropriate venue for calls to action in the community. If I were you, I would name and shame. Seriously. If it’s done politely and matter-of-factly, you have nothing to lose.
Don’t think of it as revenge; think of it as edification.
I agree with Jenny on this one – although naming and shaming can cause problems in a number of ingenious and frustrating ways, you’ve already bitten that bullet by doing so here.
I would go as far as suggesting you name and shame the editors (and if possible, referees) who let a peer-reviewed article pass through the publication process without a single one of them asking for the Methods to be included! This is atrocious. As is the Editor’s attitude (‘not our fault – the author didn’t include it and the refs didn’t notice. Life’s a lottery, eh?’).
This does a serious disservice to the peer-review process and integrity of scientific publication. Published results must be open to attempts at reproduction. Even the homeopaths gave out their methods (dilute sufficiently until only experimenter error remains) – why shouldn’t serious scientists?
Good luck with the review!
What Mike said, except the identity of the referees should not be revealed – and no editor ever would. If you review a manuscript under the rule that your report will remain anonymous, it’s not appropriate to break that pact – no matter how bad they were.
To comment on what would happen at Nature if such a mistake (on behalf of the journal, which should of course have made the authors supply the information as supplementary online methods) came to light in this way is: the journal would have responded to your complaint, Stephen, by contacting the author. If the author could not/would not respond or supply the information, the journal would, if necessary after taking advice from the same (or different) referees, publish a formal correction statement (written by the editors) noting this fact, which would be appended to the article online and which would show up on medline and other dbase searches. In this way, the authors are named and shamed, but not the referees. It is the journal that takes responsibility for what it publishes, not the referees. We are all human and we all make mistakes, whether authors, editors or referees. But when we see that a mistake has been made, it is essential that it is corrected promptly and appropriately. In my opinion, nothing would be gained by naming the referees on this occasion, so long as the journal takes responsibility for its actions.
PS, if the author had supplied the information, the journal would have published it as online supplementary information with the correction statement. Nature has done this on occasion.
Stephen, your underlying frustration is shining through. Definitely use your soapbox. I did review an article recently with less at stake, but an equally sloppy methods description. I called the authors on it, but even the associate editor seemed taken aback in the first round (the next revision was vastly better). The other reviewer barely lifted a finger. I was “raised” to believe that another scientist should be able to reproduce one’s results with the details provided in the materials and methods section. So it’s an important section to review, not least to see if you are convinced the experiments have been well conceived!
I can’t even get a supersaturated sugar solution to crystallize. Your photo is therefore very impressive.
@Jenny and Mike – I thought about naming and shaming the authors in the review (which I submitted a couple of weeks ago), but confined myself to a remark to the effect that some papers in the field lacked methods for cryo-cooling. Perhaps I should have been more up-front but in the long-run I thought it might come across as petty.
Not that this isn’t a serious issue. I very much regret not sticking to my guns with the Editor at the time (and of course, not contacting Carter back then either). I’m glad to hear, Maxine, that Nature takes this issue much more seriously. I imagine that most other journals do too. More often than not I suspect methods are incomplete through carelessness than design, but the omissions are not necessarily picked up by reviewers. It’s good to know there is a method of redress available to sharp-eyed readers.
Heather, like you, I think reproducibility is the standard to which M&M sections should be held!
Credit for the crystal and the photo should go to my postdoc, Olga; it’s of the La protein, rather than albumin. And it was tiny – the rod is only about 20 µm thick. Diffracted beautifully, though.
1. Stephen, as I read this post, I was suddenly reminded of Kendrew’s prose, though yours is better and less stiff. But there was something in the penetration, clarity, & intensity — and intent — of exposition. I do hope you’re going to write some popular science for a wider audience.
2. And that hesitation turned into a pause that lasted years. This is a lovely sentence.
3. Some time ago, on Lablit, we had a conversation about an imaginary network called GEEK-SPAN, the scientific analogue of C-SPAN. This is precisely the kind of conversation — among you all — that I think belongs in a more visible place. The public schools here are beginning to try to teach students how science works, and I think this is a beautiful exposition of how a shoebox full of — issues, for lack of better word — work together: competition, collaboration, exposition of ignorance and uncertainty; responsibility of scientists, editors and refs; careers to build and a choice of problems.
At some point, with all your permission, I’d like to use this conversation in some form in writing for students here. I’m not sure the schools will go for this much reality, but I have a stealth project in the works aimed at creating a science curriculum worth a damn, and the conversation would fit well there.
@Maxine – I agree with your point that anonymity should be guaranteed when offered, and it’s insightful to hear how Nature would have dealt with this situation. However, I’m coming to the conclusion that anonymity is not the best solution for ensuring the most thorough, even handed reviews. This probably ain’t the place for a long discourse about these things, but authors, editors and referees should all take responsibility for the quality of published literature.
@Amy – I think it’s a great idea to highlight some of these points to people looking in from outside of higher academia. To echo Maxine, we are all only human, fallible. Therefore the peer-review process is also fallible. As scientists, we should retain some healthy scepticism over all the published literature, but we do still try our hardest to understand the observable universe.
It’s good to distance ourselves from any more of a politicised message to the public.
Mike, I can guarantee you that a large numbers of referees would not write unbiased reviews if the refereeing wasn’t blind – because that assumes that people don’t take criticism personally. Recommend rejection of the paper of the guy who next year might be reviewing my grant and will therefore make or break my career? No thanks, I’ll tone done my concerns — and a suboptimal paper will get published.
Stephen, why not just reference the two papers after your comment about no methods? It doesn’t name specifically in the text, but it makes your point more forcefully. I’m sure it’s not too late to modify what you submitted.
Jenny, ever hear the one about the rock and the hard place 😉
@Amy – you are too kind. I presume you mean the crystallographer, John Kendrew? I’m familiar with his scientific output but not with his science writing. Incidentally, he only got hooked on the topic after he bumped into JD Bernal in South-East Asia during WWII.
Like Mike, I am more than happy for you to make use of this conversation for teaching about science. The aim of the post was to pull back the rug a little and see some of the dust that sometimes accumulates…!
Jenny – the question of referee anonymity is a tricky one. I guess, on balance, anonymity is to be preferred, though it would be interesting to know something about the level of abuse that derives from that. I suspect it’s not common for reviewers to act unprofessionally but haven’t seen any data beyond the anecdotal level. Do you have more of an idea, given your previous incarnation in scientific publishing?
I like your suggested modification to the review – thanks. It’s slightly less cryptic than my original – so I’ll try to remember to put it in the revised version.
Managing the peer-review process even-handedly, independently, and with expert knowledge is a considerable challenge. Nature is very happy with its system (described in detail on our website).
A good editor is more than capable of managing a fair process, and as peer-reviewers see each others’ comments when a revised manuscript is resubmitted in response to a first round of review, there are additional checks and balances in the system.
Another point to bear in mind is that scientists are authors are peer-reviewers: all the same people. Do their personalities change from Jekyll to Hyde when they are authors or referees?
@Maxine – as peer-reviewers see each others’ comments when a revised manuscript is resubmitted in response to a first round of review, there are additional checks and balances in the system.
That is a good way to push standards up but I don’t think it is adopted by all journals.
Do their personalities change from Jekyll to Hyde when they are authors or referees?
Well, Dr Jekyll was a scientist…
as peer-reviewers see each others’ comments when a revised manuscript is resubmitted in response to a first round of review, there are additional checks and balances in the system.
Maxine, do you not find that this also has a herding-towards-consensus effect? What I’ve seen in critiques of nonscientific manuscripts and student reports is that having colleagues hear each other weigh in — blind or not — does make people rethink intemperate or lackadaisical responses, but it also makes them nervous about being outliers, esp. if their own position in the group is not strong. So they’ll soften non-consensus criticism dramatically or back out altogether, but put full weight behind whatever they can agree with in main criticisms emerging from the group.
@ Stephen: That’s the Kendrew. There’s a book he wrote as companion to a BBC series called The Thread of Life — that’s the one I was thinking of.
do you not find that this also has a herding-towards-consensus effect?
I suspect this is less true of peer review than of student report marking (where some teachers may not have given the work the attention it deserves and are content to echo the opinions of others). In my own reviewing of manuscripts, I’ve never really felt much peer pressure from other reports.
Of course, I am always diligent when marking students’ output!
Amy- no. (1) people love and cherish their independence. (2) Nature is an interdisciplinary journal, so the expertise of the reviewers does not have as much overlap as it might do in a more narrow journal.
On the other hand, non-anonymous peer-review gets people into all kinds of awkward situations when it comes to revision time….and holding out for the control experiments to be done, etc.
Thanks for sharing that story.
I have heard speakers mention papers that have been published without crystallization conditions in the area of membrane proteins.
I don’t have any specific examples, but this story reminds of their similar frustration.
I guess there are many examples of that in crystallography, which is a shame. Although there used to be a trend of publishing crystallisation conditions as a separate publication, this probably harks back to the days when the structure determination was the really slow and hard bit. Nowadays, crystalllisation is more often than not the bottleneck in a crystal structure determination so if you publish these too soon, you run the risk of being scooped. These crystallisation papers have gone out of fashion a bit for that reason.
My policy would be never to publish crystallisation conditions until we published the structure. The last time I wrote a crystallisation paper (which gave more detail on how we had gone about our construct design and the particular way we had used commercial screens to find crystallisation conditions) was in 2005 and it was published after the paper on the structure came out. (I hasten to add that our report of the structure contained a full description of the methods that we had ultimately arrived at!).
BTW – although I haven’t commented there much of late, I’m still reading and enjoying your crystallography blog!
No worries, now that we are famous it is hard to have enough time to comment on all the blogs that we wish 🙂
@Ted – thanks for pointing that out! Do you think we can charge syndication fees?
Your not?
You don’t want to put Nature out of business, better keep it free for now.
I’ve noticed that in many of the discussions on this network, such as this one and here , the word ‘referee’ is nearly always used by most people for the person an editor asks to assess a manuscript for them. As a ‘referee’ is (Oxford) “an umpire, a person whose opinion or judgement is sought in a dispute”, is it the appropriate word to describe what this person does in the peer-review process? Rather, they assess and advise, and it’s the editor who makes the decisions. I always therefore use ‘reviewer’, and change referee to that in all our editorial correspondence. But I think I’m in the minority (along with a few others ) as there is such a long tradition of using ‘referee’ and ‘refereeing’, and maybe people no longer think about what the words mean. Can I convince others to switch to ‘reviewer’ and ‘reviewing’?
I couldn’t agree with you more, Stephen, about the importance of the Materials and Methods. Some reviewers even look at those first. And we as a journal would act as Maxine describes above (26 April) if someone brought to our attention that a piece of work we’d published had inadequate details about how it was done and the techniques and materials used.
Thanks Irene! I suspect that Brian Derby and I may simply be showing our age when referring to reviewers as referees. I can’t recall how I picked up the habit but, on reflection, I think I am bound to agree with you that reviewer is a better choice. And who am I to argue with the person who has written the book on this!
Very glad also to hear that your journal is among the many that pay proper attention to Materials and Methods.
Mea culpa too, Irene. I try to remember to call them reviewers (though can be confused with “authors of literature review articles” so I try to use peer-reviewers) but I forget more often than I remember.
BTW, thanks for the link, Maxine. À propos the title, I’m surprised more people haven’t noticed the connection with Fagin’s song from Oliver! Perhaps they’re just much more high brow than wot I is.
Maybe not as many of them have been forced to, if not sit through, then hear in the background, several thousand attempted renditions in that awful talent show “You can be Nancy”, or whatever it was called. As a result, the songs in that musical have a kind of horrific resonance whenever one comes across an allusion to them.
That talent show format, by the way, is another form of peer-review, come to think of it….
I’m not generally a fan of the musical as an art from, apart from Oliver! (which seems to have a childhood resonance not spoiled for me by the Nancy-mania since I avoided it), West Side Story and Guys & Dolls.
But these TV talent shows don’t have much peer-review since it is the sometimes talented being judged by the largely talentless population. My daughter somewhat acidly remarked that ITV’s flagship should, barring one or two exceptions, be renamed “Britain’s not got talent.”. Such cynicism in one so young. I’d beat it out of her but I haven’t got the heart.
Thanks, Stephen – glad to have a convert!
@Jenny (from way up): Stephen, why not just reference the two papers after your comment about no methods? It doesn’t name specifically in the text, but it makes your point more forcefully. I’m sure it’s not too late to modify what you submitted.
Got word this morning that the review has been accepted. No modifications were requested by the reviewers but I took your advice and made it clearer where the faults of methods omission occurred. So thanks. What a Web2.0 moment!