Well, now that all the excitement‘s dying down (for now), I finally find myself with enough time and motivation to start writing a follow-up post to the inaugural Science Online Vancouver event, which I attended on a very rainy Thursday night last month.
I’d read a lot about Science Online events in London and New York, and was exceedingly excited about Vancouver getting its own version. The first event did not disappoint, despite the unfortunate lack of food or beer for purchase (I think I’ve been spoiled by our local Cafe Scientifique events, which always take place in a pub, as compared to SOVan’s choice of the Science World museum).
The theme was “where do you get your science?”, and we started off with a presentation from UBC researcher and one of Nature‘s ten most influential people of 2011, Rosie Redfield. I’d seen Rosie talk before at a Cafe Scientifique event about evolution, but this time she was talking about her role in debunking the infamous Arsenic Life SNAFU. I’d read a lot about the case at the time, and it was great to hear the story from Rosie herself. Next up was local reporter Lisa Johnson, who covered the original story and some of the follow-up for the CBC. She told us about the hype generated by NASA and the time pressures involved in getting the story out – again, extremely interesting to hear direct from the source.
One of the themes covered in the subsequent discussion – and the main point of this post – concerned the practice of Open Science. Rosie Redfield, who’s been using her blog as an online lab book for years (and described the Arsenic Life saga as her first opportunity to do so “with people watching”), made some compelling arguments in favour of full openness. I can’t argue with the fact that most people don’t know how science is really done and that letting them see the reality is A Good Thing, in principle – nor with the sentiment behind this statement, which many people picked up on and tweeted:
[blackbirdpie url=”https://twitter.com/#!/polarisdotca/status/193173391384117249″]
However, I felt that I did need to speak tweet up with my own experience…
[blackbirdpie url=”https://twitter.com/#!/enniscath/status/193173196252516353″]
(I don’t know why those tweets have a British time zone date and time stamp, btw. This took place on the evening of April 19th, not at 3 am on the 20th…)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
(Cue cheesy flashback harp music…)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
It happened just over a year into my postdoc, when I was writing up my first paper. I’d discovered that the transcription (activity) level of a human gene is controlled by a piece of DNA (called a promoter) derived from an endogenous retrovirus; characterised the promoter to find out which proteins were binding to it to control the activity of the associated gene; identified a novel alternative promoter that seems to pre-date the insertion of the virus; and compared the activity of both promoters in a number of different human tissues. I was about a week or two away from finalising a submission-ready version of my manuscript when I got scooped by a paper that showed up as a pre-print publication in the online version of the journal to which I was planning to submit. Using a very similar approach to my own, they’d identified some of the same promoter-binding proteins and done similar analyses of the promoter’s activity in different tissues – but hadn’t spotted that the promoter was derived from a virus, and hadn’t looked for the alternative (original) promoter. So it was a partial scoop – around 75-80%, I’d say – but still a huge blow.
I ran straight to my supervisor, who quickly talked me down from my panic and said that we’d “just quickly reformat it” and send it to a different journal (one with a higher impact factor than the one we’d originally targetted, funnily enough, not that she cares much about such things). Of course, it wasn’t quite as simple as that – it took a few days to reformat the text and figures, and we had to cut a LOT of text to get it down to the right length – but we got it done, with help from sympathetic lab mates, and sent it off with bated breath.
I got lucky. The scooping paper hit the print edition just before we got the reviews back (print vs. online was a bigger deal then than it is now!), and both reviewers had clearly read it. However, because our paper was submitted just a few days after the first one came out online it was clearly independent work, and had enough unique findings (and enough of a different focus in the introduction and discussion, i.e. on the evolutionary implications of the endogenous retrovirus vs. native promoter rather than on the function of the gene itself, which was the source of the other group’s interest in the promoter) that it was accepted with minor textual revisions, including a requirement that we mention the scooping paper and discuss it very briefly. I even managed to turn this into a positive by pointing out that they’d identified the same promoter-binding proteins that I had…
…I got lucky.
A year or so later, a good friend of mine and fellow postdoc in the same lab was not so lucky. He’d been working hard for a long time to set up and validate a system in which to test a hypothesis about the function of a different kind of endogenous retrovirus, and was getting very close to being able to start his actual experiments when the big shot lab in the field scooped him completely. They’d even used an almost identical system. He was understandably devastated; when our supervisor gave him the bad news, he pretty much ran straight out of her office and wasn’t seen again for the rest of the day. He did have some other side-projects running, and was a co-author on a couple of other papers from the lab, but that had been his main project and it was suddenly completely unpublishable. It really was essentially the same study.
(A third postdoc in our lab also got scooped at around the same time, but I almost hesitate to mention it due to her weird response. I saw the scooping presentation at a conference and gave her the bad news myself as gently as I could when I returned to the lab, remembering my panic from the previous year all too well. She said “ah, well, thanks for letting me know!”, and just got on with whatever the hell it was she was doing. Not working, that’s for sure. She was already spending more and more time “working from home” by then, although no evidence of any work was ever forthcoming at lab meetings, which she eventually stopped attending before dropping out completely to join an organisation that sues people who call it a cult, so I won’t do that. It rhymes with Sand Bark Zoundation. I will not link to it. But I digress).
~~~~~~~~~~~~~~~~~~~~~~~~
(End flashback. More harp music)
~~~~~~~~~~~~~~~~~~~~~~~~
So, while I do understand the very admirable and altruistic impulses behind the Open Science movement, I find it very hard to commit to the cause. As long as careers depend on publications and publications depend on novelty, scooping will remain a major problem in competitive fields, a problem with the potential to completely derail young scientists’ careers. As I said, I got lucky – it wasn’t a full scoop, and I was so close to submitting the manuscript that the work was obviously independent – but my friend’s experience makes me exceedingly cautious. The risk of scooping is field-dependent, of course, and a case could be made for opening up your lab books only once the work reaches a certain stage, but sorry – I can’t get on board. Not really, not in the current scientific publication and careers paradigm.
Anyway… Science Online Vancouver really was awesome. The follow-up discussions were all very interesting, and I met some fabulous people (most of whom seemed to know at least one person I know – as I’ve said before, I swear the Vancouver life sciences community only has two degrees of separation). A few of us went to the pub after the main event ended, but wet feet and jeans, a lack of food, and having to get up in the morning for work sadly sent me home after only one beer, and before I could introduce myself to some of the people I’d wanted to meet. Clearly I just can’t keep up with the young ‘uns any more… but hey, there’ll be more events!
Surely if everyone was being open these would never have happened, because they would have talked / noticed each other before doing the experiments?
my sympathies – it’s bad news being scooped or pre-empted to publication, but this kind of stuff has been happening in the science world since time immemorial; eager grads and doctoral students presenting posters at conferences risked having their posters photographed and work duplicated before the online, digital age. there’s always the weird coincidence factor which comes into play occasionally which may or may not show a breadcrumb trail back to the ‘leak’.
i may have missed this in your post, but did you post about your research online before writing up for publication or is it more of a cautionary tale? did you have any kind of trail to follow back to the leak?
In the C. elegans world, there was for many years (and maybe it still exists now) a newsletter type of journal called The Worm Breeder’s Gazette. People would submit all kinds of unpublished things to it, in a format more or less like meeting abstracts. It was not, of course, peer reviewed. It served as a community publication so that people could know what everyone else was doing and spur discussion and collaboration, in the best spirit of the worm community, which has always been a very collegial bunch (in general).
Sounds great, right? Well, it was… to a point. It also, of course, served as a sort of virtual pissing post, so that people could essentially stake their claim to certain research areas and implicitly discourage others (or, if you like, encourage them to do something else). Yes, it kept people from duplicating efforts, thereby avoiding scoops. But it also meant that those researchers not playing the WBG game could sit back on the sidelines and keep an eye on what their competitors were (quite publicly) up to.
So – as an “open community” approach – same old same old. If everyone plays nicely, it’s great. But we all know that science doesn’t work that way.
Last related point – there are many examples where full-on competition has sped things up (public Human Genome Project vs. Celera; the Duchenne Muscular Dystrophy, CF and retinoblastoma genes, structure of DNA, you name it). It doesn’t stretch the imagination much to think that fully open science could actually slow a lot of discoveries down, rather than speed them up.
I’ve always understood the ‘Open Science’ movement meant publishing your data along with your results. Obviously some people want to go further, with open lab-books, but I think that’s probably a step too far, risking encouraging and rewarding the wrong sort of behaviour from people who haven’t collected and don’t necessarily understand the data.
The same argument might also apply to data published with a manuscript, but a proper manuscript should outline all the important details anyway.
But I would say that, I collect/generate very little (or no) primary data. And I’m an OpenScientologist.
Adam, welcome to the blog! I applaud your idea in principle, but I don’t see how you’d make sure that everyone starts being open at the same time. Also, you’d have to watch so many other groups – the second scoop I described (the really bad one) came from a group we’d have been keeping an eye on if we’d had a way to do so (they had a habit of registering for conferences, but then you’d go to their poster or talk to find a notice that they’d withdrawn at the last minute – you’d never got those guys to sign up for open science in a million years), but the first and third came from labs we’d never heard of who were working in completely different fields and converged on the same problem as us from another direction.
Pomomama, welcome to you, too! I didn’t post about my scooped work online before publication, but I had presented it as a talk at a conference just a couple of months before the scoop – so that wasn’t why it happened, because there wasn’t enough time in between (also, as I said the group was working on very different problems and was very unlikely to have been at the genetics/evolution-themed conference at which I presented. They were biochemists who were looking at the gene’s promoter because they were interested in the gene’s function, whereas my group was interested in how the promoter itself had evolved). I really don’t think there was any leak or any kind of shenanigans at all, it was just a genuine coincidence of two groups approaching the same problem from different directions. The third scoop (the one where the scooped postdoc just didn’t care) was much the same – I don’t want to say it was an obvious thing to look at, but it wasn’t exactly a wild and crazy left-field hypothesis, and the technology to look at it had just advanced far enough to make it practical.
The second scoop, though – the really bad one – that was weird. It seemed like far too much of a coincidence that they’d chosen the exact same examples of each type of virus to look at, the exact same plasmid to clone things into… This was the group I mentioned in response to Adam who were extreeeeeemely secretive about their own work and never responded to any attempts to engage with them, which does tend to make you wonder.
My real intention, though, was to show that scooping does happen and it can have disastrous consequences, especially for early career scientists. It’s not just a hypothetical big bad wolf or bogeyman dreamed up by paranoid people – it’s a reality.
Richard, I’d heard that “worm people” are great like that – very collaborative. But it does sound like the system is misused by the territory markers (for how long can you play that game of dissuading others from doing their own work in that field, and in how broad an area?) and the lurkers.
Great point on the benefits of competition (in some cases).
Mike, there may well be more than one definition, but the one Rosie Redfield was talking about was the open lab book model (see her blog for examples of this kind of open science in practice).
“Richard, I’d heard that “worm people” are great like that – very collaborative. But it does sound like the system is misused by the territory markers (for how long can you play that game of dissuading others from doing their own work in that field, and in how broad an area?) and the lurkers.”
For whatever it’s worth, I’ve often found myself perceiving some of those throw-away remarks at the end of a published paper to the effect the team are investigating such-and-such as follow-on work to be to a similar ‘territory marking’ end.
I offer more examples of being scooped and related issues, but I think I might do that in private email! – potential repercussions and all that…
Adam, I’m sure that is true in some fantasy world where everyone has everything in the open. And also there is no one that would see your “open” work and instead of being all warm and fuzzy collaborative decides to speed up their work in the same area to beat you to the punch. But in the real world, that is not how it works. It seems ridiculous to ignore that. Do you want to be the one that sacrifices their ability to publish (and perhaps your career)?
Open science is an interesting idea, but I’m with Cath here–I can’t really get on board. I’m a junior faculty member, so I need to publish in order to keep my job. And my grad students and postdocs need to publish to advance in the careers that they want.
exactly – you absolutely have to protect your own interests. I hate that it has to be that way (and that I’ve apparently become such a cynic), but that’s the way it is.
I don’t think being scooped has much to do with open science, to be honest – I agree with pomomama. I have been working at Your Favourite Etcetera since before the interwebz, and people were at it then.
In almost every case I know of where one group scoops another, each group could have benefited from the collaboration. If you think about it, the only way this can’t be the the case is if the final papers contain exactly the same data. Is it really a good strategy to fund two, or more, labs to do the same research? In my mind, it’s a no-brainer that the cause of science would be advanced greatly if scientists worked together.
Of course, it sucks to get scooped, and no one working on p53 in HeLa cells would be wise to do it. If the field began to recognize the contributions that each scientist made to a project, instead of characterizing them simply as “first author”, “senior author”, or “may-have-done-slightly-less-work-than-first-author-or-may-have-just-done-a-couple-ligations” it would help a lot.
Casey
sorry, our comments crossed. Welcome to the blog! Lots of first-time commenters today 🙂
Yeah, it would definitely be nice to have a better way of crediting contributions. Right now, giving up first author on a paper from your own group in order to be second or the controversial joint-first author on a collaborative paper that lists twice as many authors is not a good career move, whatever the other benefits of collaboration may be.
*that* explains how all those other comments showed up after I posted mine!! I was a little confused 🙂
yeah, all the first-timers’ comments are being held for moderation. I’m approving them as fast as I can, work permitting!
gerty – for what it’s worth, it’s a fairly common anti-spam measure; I do the same at my blog.
Casey,
An early post of mine tackled this issue. (My ideas have shifted slightly and I cringe at parts of it now – the original is from ~10 years ago – but it’s still food for thought perhaps.) I think a ‘full’ solution to this is quite tricky in practice as different fields have different issues and the bibliographical databases will want standardised meta-data. An interim thing some so is to add a footnote to the paper clarifying contributions.
Some great ideas in that post, Grant! I agree that the current system does not accurately reflect effort, neither quantitatively nor qualitatively.
I’m not saying that open science is the only cause of scooping; I’m saying that the existence of scooping makes open science an impractical idea. One of the arguments at Science Online Vancouver was that scooping isn’t a real threat, but that is most definitely not my experience.
that was in reply to cromacrox
There’s a another issue here, which is the underlying reason why “scooping” is a problem.
We place too much priority on being first and not enough on replication.
You would think that two labs arriving at the same conclusions independently should be reason for both papers to be published as complementary findings. It would show that the finding was rock solid.
The arsenic life story was also a good example of how completely ambivalent people are to replication, even of a big, high profile claim.
I’ve blogged about this (I find myself saying that more and more these days – am I running out of ideas?) here : http://neurodojo.blogspot.com/2011/06/what-coburn-report-has-in-common-with.html
What Zen said -replication is important, and underestimated.
If two labs do arrive at something simultaneously, so much the better. Less chance it’s wrong.
It is also a major way fraud is uncovered, I would have said.
Re ‘too much priority on being first’ I once had the experience of publishing a idea nearly at the same time as another lab. One of our more bitter professional rivals took exception to our later being credited with ‘simultaneous but independent’ in a commentary, and wrote to a journal editor (!) who’d published the commentary, pointing out our paper had been submitted a month later than the other discoverer’s. Coincidentally, other discoverer and bitter rival were friends.
I agree, and I think there’s a common theme here with Casey’s and Grant’s comments on how scientific contributions are credited. The current system clearly just doesn’t cover all the different types of contributions that are needed in order to do good, solid science. This is definitely something worth fighting for – much more so than open science, in my opinion – but again, any push really needs to come from the top; early career scientists should not have to martyr their own chances in order to make this happen.
I’d like to point you at my blog post from a few weeks ago:
http://ivory.idyll.org/blog/apr-12/blog-practicing-open-science.html
Scooping, in my case, was and is a background concern… but definitely there.
–titus
Welcome to the blog!
Very interesting post.
“This is a conundrum for open science: how do you know that someone else’s work is worth your attention? Research is really hard, and it may take months or years to nail down all the details; do you really want to invest significant time or effort in someone else’s research before that’s done? And when they are done — well, that’s when they submit it for publication, so you might as well just read that first!”
That’s a good point, but field-dependent, I think. I’m not a bioinformatician (although I work with lots of them), but I’ve picked up enough information about the field to understand about not wasting your time on prematurely-released buggy code. In other fields, though – say, molecular biology – the results of a single Western blot or similar experiment could be just as informative and useful as anything included in a peer-reviewed paper, if released with enough experimental information and the relevant controls etc.
I wonder if the usefulness of a fragment of a mature study as a stand-alone entity correlates with the likelihood of it being scooped? 😉
As has been insinuated in previous comments, I’ve heard some people voice the opinion that adopting the principles of “open science” might actually be a protective mechanism against scoopage — perhaps as a way of proving that you were working on something before and independently of another group’s publication. I have no idea how much weight this would carry with journal editors, but perhaps it could be used in support of a request to publish back-to-back papers?
It seems to me that people who are determined to be unscrupulous already have ways to get ahold of their competitors’ data even without it being available online before publication; so in a sense, the only difference might be in the increased forging of collaborations between decent people.
One caveat to “open science” that is seldom discussed is the importance of achieving intra-group consensus on this issue (and the consequent difficulty it poses to individuals wishing to adopt this model in practice, even if they fully back it in principle); I wouldn’t dream of posting any of my findings online because they rely heavily on other group members’ unpublished data.
I would be very interested to know how the extent of sharing pre-publication data varies between fields; anecdotally, I’ve heard that biomedical research is notoriously competitive, and that at least some subfields within maths/computer science/physics/engineering are very collaborative.
Also, my gut feeling is that people who are most heavily invested into their projects have the least to lose from posting the data online before submitting to a journal; if your experiments take years to do, it is very unlikely that someone could replicate them before you submit yours for publication…
Some great points, SB!
“I have no idea how much weight this would carry with journal editors, but perhaps it could be used in support of a request to publish back-to-back papers?”
maybe… perhaps Henry or someone else who works in that field can chip in here? My feeling is that under the current system it wouldn’t carry much weight at all, if any. “Hey! We were working on that first! Sure, we haven’t actually finished the study yet, like these other guys have, but we marked that territory!”
I wonder if some labs are split down the middle on open science? I’m guessing that if the PI is absolutely 100% dedicated to the cause then they let trainees know before they sign on in that lab (one would hope), but in most labs I know that weren’t specifically recruited with that criterion in mind, there would likely be a split.
The issue of scooping relates to the old secretive publication model. If we adopt the arXiv or Figshare post-publication peer review models, then the scooping problem goes away. Seems to me your understandable concerns relate to not enough open-ness, rather than too much.
From what I’ve heard of it (mostly from my physicist friends), the arXiv model works fantastically well in some fields – but those are fields that have matured around that model. If you want to introduce the model to a more secretive field like biology, the implementation will be hugely problematic. As a few people have said, every system will contain some cheaters/free loaders as well as the honest/worker bee types – so unless the new model is made mandatory for all, there will be elements that take advantage.
Like many others above, I don’t see how openness makes scooping worse. Putting the blame on Open Science seems to be a case of shooting the messenger.
Would it have been better if, instead of noticing the scooping during the pre-print, you found out after publication?
Anything that makes things easier for the less-than-optimally-honest element will make things worse for the others.
The only difference the pre-print rather than full publication status of the scooping paper made was that it gave me slightly more time to get the paper re-formatted and submitted elsewhere than if I’d seen the scooping paper in print rather than online…
The argument about open science and the subscription journals is like the argument between open source code movement against proprietary programs. Do you blame those who release their source code when they are copied by the proprietary programs?
I wouldn’t say blame, but I’d think they were being somewhat naive.
Do you blame people who post their credit card numbers online when they get ripped off?