I’m afraid to say it out loud, but my lab notebook just isn’t up to the job anymore. Last time I talked about how difficult it is to display some forms of modern experimental data in a readily comprehensible fashion. But that’s not the only problem I’m having these days. Documenting my work in the trusty notebook is also growing more futile: most of my data consists of monster spreadsheets and terabytes of images and videos. I know journals have overcome this problem by exploiting online publication and supplementary data, and labs, by creating vast storage databases on their websites. But gone, for me, at the personal level, is the ability to record everything that I am doing with a pen, on paper.
This state of affairs is particularly distressing for someone of my temperament. In an older post, I explained how I am a compulsive documenter when it comes to my experiments: no piece of film, for example, is blank enough to escape my scissors and tape; no failed PCR gel is too smeary and inconclusive. I like, in short, to record every detail from triumphant eureka to notorious bellyflop, including scribbles, images, graphs and charts, snippets of email printouts from collaborators – more like a geeky scrapbook than the sort of documents a patent judge might want to subpoena.
Now, my attempts at summary grow increasingly half-hearted – most of my book is just an index, pointing to a series of files on DVD and a growing family of external hard-drives. Given the fragmented nature of the narrative, even I have problems following the logic of my activities some days. Worse, I don’t really trust DVDs and hard-drives; I’ve had enough of these spontaneously corrupt to know that I can’t rely on their permanence.
But then, it was ever thus: the ways and means of science have been briskly evolving since I entered the research game back in the late 1980s. My year in graduate school was the last group of students to paste photographs directly into their Ph.D. theses; flipping through it now, I marvel as the pages fan by, weighed down by Kodak paper and glue. I recall, too, the stab of jealousy I felt when the next year’s crop of students showed off their magna opera, all images neatly scanned and incorporated into the document. I’ve seen the conversion from slides to PowerPoint, and the PowerPoint fads come and go: yellow text on a fading gradient of dark blue; cheesy animation transitions; that entire grim year when Comic Sans was the only font you ever saw at American conferences.
Things change, and I’m going to have to learn to live with my stripped down, new-age lab journal. But I do confess, I won’t be able to love it quite as much as before.
There are wibblings in the biosphere about electronic notebooks. No idea how well they’d react to sitting on my bench however, sandwiched between the phenol and the ‘DNA Precipitation Buffer’.
Not to mention we’ll lose the stories told by coffee stains and smudged ink.
Still not sure how that would help me tell the story of an experiment that (say) yields 2000 images; display and documentation will still be very difficult. Storage size, too, has got to be an issue.
And you’re right – mine would be shorted out by a lab mishap in no time.
But at least you’d have a direct link between the lab-side notes and the electronic data that are on a server somewhere. You’d obviously need to sync with a computer to bring everything together in a sensible form.
Hey, you’re right – a quick google of ‘electronc lab notebooks’ has yielded a number of different products on the market. But I’ve never heard of a real-life scientist actually using one. I’d be interested to hear from anyone who’s seen one of these babies in action.
I’ve recently begun using Google documents as my lab notebook. However, it contains (as of now) only descriptions of the experiments I have performed, not the data I get from it. Most of the data is quantitative, so I have a lot of excel spreadsheets. But, for now on, I think I’m going to use Google spreadsheet. Once I get all the kinks worked out, I’ll probably write a blog post about it.
Jenny, I think you’ve hit on a real problem. As experiments yield more and more data, it must be ever more difficult to follow the essential thread of an experiment, from hypothesis to test to conclusion – to pin that process on to the morass of data themselves. What’s more, it will become more and more difficult for other people – people who haven’t done the experiment, who don’t have a kind of model of it in their heads, who’ll be coming to the process from cold – to work out what’s going on, too. I am beginning to see the signs from my side of the fence. Referees who are starting to whimper at the sheer volume of supplementary information appended to manuscripts, and wondering how on earth they are going to establish that the experimenter’s path between hypothesis and conclusion doesn’t have some tiny logical flaw, somewhere, buried among the data. Even palaeontology is not immune: these days it’s become very data-rich, with huge animation files, CT scanes and so on. Sometimes one yearns for simpler days.
Brain doping. Obviously.
Why not get all Web2.0 and turn a blog into your lab notes. You wouldn’t have to publish it, just have it on a local drive (networked?), and then you can link to all your data files.
If you want to do an impression of someone who is organised, then you could create a wiki to describe your projects.
I should point out that if I tried something like this it would barely last a month. I’m slightly surprised my blogging has lasted so long.
Points
I love it when the elves have been hard at work while I sleep.
@Nuruddeen – First I’ve heard of these Google products; definitely worth a look (for the spreadsheet function alone). It probably doesn’t solve my all my problems, but I’m keen to see it nonetheless.
@Bob – Blog as lab notebook – I love it! I suspect it wouldn’t work for me but it’s a nice thought. And can you imagine, all one’s colleagues leaving catty comments? (“I wouldn’t have done it like that…”)
@Henry – Very interesting that journals too are feeling the archival heat. Where will it all end, I wonder?
It wouldn’t have to be a public blog – you could stop anyone else from reading it! or just the immediate lab.
I feel a blog post coming on. But I have to finish off 8421 Canadian sheep first. This may take some time.
I suspect (hope) that you will soon be seeing comments from Cameron Neylon and Jean-Claude Bradley; both active at Nature Network and proponents of electronic lab notebooks. In the meantime, here are relevant posts at Cameron’s blog, Jean-Claude’s UsefulChem blog blog and the UsefulChem wiki.
Note that these guys are proponents of open notebook science but as Bob says, blogs don’t have to be public.
See also Rosie Redfield’s group, where everyone is encouraged to blog about their experiments.
A web search for “electronic laboratory notebook” will throw up plenty of material, but not much help. There are commercial products available, more suited to industrial labs with tightly-defined workflows (and big software budgets!) Many people are finding that blogs, wikis or a combination (“blikis” – yes, really) work very well as notebooks. They give you many of the features that you need: timestamps, upload of various media and file types, revision control, hyperlinks to connect your data. However, you’re just not going to find a one-size-fits-all prebuilt notebook solution.
So again, it’s time to start connecting with your friendly, knowledgeable NN informatics enthusiasts!
Thanks Neil. I look forward to diving into those links! (But open lab notebooks? You’ve got to be kidding. These guys have obviously never been scooped. This would never, in a million zillion years, work in any cutthroat field.)
Blogging about your experiments to your labmates, when they are all around you, in the flesh? That seems a bit isolating. Whatever happened to good old-fashioned conversation?
And as for you, Bob: Re: “But I have to finish off 8421 Canadian sheep first.”; I’m afraid I don’t even want to know what you meant by this! Sounds either like kinky dealings, or an episode of The Archers.
Neil has hit the hail on the head. Commercial products currently on offer are designed for commercial labs.
There is interest in the academic world, but these tend to be projects and not necessarily generalisable. JISC have a nunmber of projects in their Virtual Research Environment programme – http://www.jisc.ac.uk/whatwedo/programmes/programme_vre.aspx
My impression of a couple was that they were designed for big collaborative (e.g. EU-funded) projects where there were a number of labs needing to share the same methods and results.
VRE / ELN solutions are also of interest to research funders as the first step on the road to capturing/storing/preserving research data.
Frank, the international sharing imperative is an angle that hadn’t occurred to me but which is bound to become increasingly pressing. But again, is the lab journal really the right level for this? My labmates can’t even understand my shorthand – not sure a collaborator could make heads or tails of what is essentially a messy brain-dump that will need to be extensively tidied later on.
My labmates can’t even understand my shorthand
you’d better not discover anything patentable then…
Been there, done that.
To clarify…my notes have been useful for the patent adjudications (more than most, apparently, due to the level of detail), but this is because I am quite good with intros and summaries of data. A lot of the process gets short-handy, but apparently it’s like this for everyone. It’s precisely this logical narration I’m worried about losing.
I just googled VRE. Apparently it’s either “Virginia Railway Express” or “Vancomycin-resistant enterococcus”. And ELN is the National Liberation Army of either Bolivia or Colombia.
Dangerous stuff this eResearch.
@Jennifer:
Blogging about your experiments to your labmates, when they are all around you, in the flesh?
I agree, that would be pointless – but the point of the blog is to be your lab notebook. A searchable digital archive that you and your labmates can use as a resource? Sounds a bit more promising.
These guys have obviously never been scooped
Jean-Claude has some interesting ideas about this. Basically, if your data are timestamped and clearly owned by you, how can you be scooped? It’s like trying to steal a famous painting. I see your concerns though and perhaps it varies with competitiveness of the field. But as I said, electronic notebook science doesn’t have to be open notebook science. Your blog/wiki/bliki/whatever can be as open or closed as you like.
I left the computer to deal with the 8421 sheep and it’s taking a long time, even after I removed their sense of identity. So, I’ve blogged at far too much length about virtual tools. It’s an encouragement for someone who actually knows about how VLEs, VREs etc work in practice to show off their knowledge.
Jennifer – those narratives are absolutely essential. Raw data – whether kilobytes or terabytes – is very little use without something that explains what it is, how it was obtained, etc etc. Just as The Archers is so much more enjoyable when you know all about Shulagh’s past history and Jennifer’s various love-interests.
Trouble is, you (the investigator) are the only person who can create the narrative.
I have no experience of VREs etc but my assumption would be that they facilitate collection of teh narrative bits, not just data.
Neil, regarding your comment ‘Basically, if your data are timestamped and clearly owned by you, how can you be scooped?’
A lab can always claim that an idea arose independently – for example, no one can prove that I was snooping around your notebook; my brilliant Nobel-winning paper could have been inspired entirely from my own thoughts (even though I stole the idea from you).
We know this happens because people have been known to give public talks and posters, thereby inadvertently giving away their ideas too soon to a watchful competitor. This is why people are wary of publicly presenting data prematurely, and this is why open notebooks are probably never going to catch on. In my view, anyway.
@Jennifer – not my words, I was paraphrasing Jean-Claude Bradley. More of his thoughts are in this thought-provoking SciAm article.
Anyway, let’s not get sidetracked into open notebook science, it’s not for everyone. What we’re after is a digital solution for you 🙂
True! Sorry, I’m always getting side-tracked by the messy humanity of scientists in their natural habitat.
It’s interesting to see how often this meme has come up, both on the blogosphere and in lab conversations that I’ve heard about. Neil’s pretty much reflected my point of view. The key part is that you need to have a time stamped record, with edit history, and the ability to query, mine and search. This doesn’t have to be open, although that’s a good thing in many cases. Think 37Signals
Thanks, Deepak. I don’t know how this keeps happening: I blog about my life in the lab and end up tripping over more bioinformaticists than I have ever met in real life. This either indicates something about me that I don’t understand – or else that bioinformaticists secretly rule the world.
There’s a lot to chew on here.
I wrote my thesis about 7 years ago and I don’t think I would have been able to use an electronic notebook: I need to write doodles ! doodles ! doodles ! doodles everywhere ! 🙂
However, nowadays, I use a wiki on our intranet: this is very useful to record what I did and how I did my daily job. Everyone can then retrieve what/how I did my work. I also thought about using a tool like deli.cio.us/www.connotea.org which would work on local files (e.g. adding tags to an scan) (for example http://www.chipx86.com/wiki/Leaftag (never tested) )
I’m also thinking about Attila Csordas who’s writing his thesis on the web http://pimm.wordpress.com/thesis-live/
Greetings, my white-steeded hero.
I confess I can’t parse this sentence at all: “I also thought about using a tool like deli.cio.us/www.connotea.org which would work on local files (e.g. adding tags to an scan) (for example http://www.chipx86.com/wiki/Leaftag (never tested) )”
Such are the divisions between your understanding and mine.
Ah well, vivé la difference.
Wierd, would have responded to this ages ago but couldn’t get on at all today. errmmm what they said. Blogs are a great way to capture things along the way and try to impose a bit of order on them. Wikis are a better way to organise things after the event.
And the thing about it not being worth doing it for your lab because they are already there is a furphy. Anyone who has put a bunch of protocols on a lab server and then had everyone else use them after that knows that providing useful data in a slightly more permanent form is very popular.
Yes, in my group we make it completely open. But it still remains the fact that the person who benefits most from this is me, because I can find things. The scooping thing…well I’ve written on it here and here so I won’t just recapitulate the arguement there. I will finish with a comment that I think sums something up, this from a physicist, I can’t remember who exactly.
‘You biologists are so scared of getting scooped that you sit on your hands waiting 12 months for your work to come out in a journal. Us physicists look at you and wonder what a slow moving field you must be working in that you can afford to wait that long for your results to come out’
You know this reminds me about one thing that someone mentioned the other day. At a lot of software companies developers maintain blos (on the intranet). The blogs come in use during annual review, to get tips from developers in other groups etc. To an extent that’s one of the bigger uses of lab notebooks. On the other hand, just that scope is limiting as well.
Too many comments to parse here! In terms of existing commercial ELNs I am unconvinced of their value in an academic lab setting. They tend to require lots of technical intervention to get them working well in a particular commercial setting (with multimillion pound roll outs) so they aren’t easily applicable in a cash strapped lab.
In many ways a word document with hyperlinks to a good directory structure is a perfectly good approach which works for lots of people. So does a simple html page. The key question is what you want to actually use this ‘Lab Notebook’ for. Is it just an index, or is it a collection of your thoughts and ideas. If its all electronic do those need to be in the same place?
The key driver in figuring out what you want should be what you are actually going to do with it. This will evolve so flexibility is good but keep focussed on your needs. That’s what is important.
Cameron, that thing about waiting for results to be published is something that exercises me ( here for example). Why can’t we do this better than the physicists?
After all, we have a secret
clone armyarmy of bioinformaticians sitting around with nothing better to do.@deepak: Shirley Wu also made the point that in some ways the use of ELNs in commercial pharma is a great example of the social effects of open notebooks. All right we don’t get to see them but the big guys are actually calculating how many millions they are saving by letting everyone within the organisation see everyone else’s lab book.
Internal sharing is still very useful and effective. It certainly works for us with collaborators where you can just point at a post and say, ‘thats what we’ve sent you’.
@Cameron – just to gently point out that while our papers are in press, we are busy sorting on the next three papers; nobody is sitting on anyone’s hands. Also, when a paper is in press, scientists actually can relax and talk about it – the moment of being scooped has past. Finally, it’s all very well to prioritize communication about preliminary work over paper publication, but biologists cannot survive without papers. I’d rather keep unsociably shtum than risk losing a grant – or worse, a job offer or promotion, because someone else in the free-and-easy Web 2.0 world has helped himself to my ideas.
So ELNs are only for the rich and famous? I’m disappointed.
p.s. Frank: Good God, is that how you spell “Shulagh”? What sort of language is that?
@Richard. Well I guess our friendly hosts here would point us to precedings as a starting point. But I don’t know why there is such a divide on preprint publication. The chemists are even more anti- (Amercian Chemical Society journals are some of the few that don’t allow any pre-publication at all).
But its tied up with the notion that nothing is ‘real’ until it’s peer reviewed. And that papers are the only contribution you can make. And that the submission date is the only marker that counts. Which is no use if, as you know, you go around the houses for 12 months until you find a journal with sufficiently enlightened referees to actually understand what your point is.
But now I’m descending into my ‘generic open rant’ 🙂
@Cameron: Re “But its tied up with the notion that nothing is ‘real’ until it’s peer reviewed.”
I was a journal editor for four years recently, and I can soundly attest that many manuscripts don’t pass peer review because they aren’t real. Some are 100% not real, some are part unreal, and some are just plain surreal.
We still need a filter, IMHO. But then, as I’ve said before, I’m famously old-fashioned.
@Jennifer: I agree but the 6 months it takes between getting the data and writing the paper and the next 6 months waiting for it to be accepted (if you get into the first journal) is a long time for your work to be sitting around. Other people could be using it advance other work. Or perhaps not just replicating it?
And the whole point of the open agenda is that people can’t steal your ideas. You’ve already laid your claim to it. And if you’re really clever and someone comes along and flogs it, you’ve even got a record of them having been there 🙂
I know I’m a bit utopian, and that I have the luxury of being able to be so. But I do actually believe that we could do far better science with far higher efficiency, and far fewer human casualties, if we were more open than we are. I do know that there are practical (and ethical) limits for many people but it bothers me that so much of the way we do science is driven by fear rather than excitement.
I don’t disagree with the need for filters. I’m just convinced that it would help to have a bit more variety in how they work. That’s a whole other argument though…[time for me to be heading off. One thing which is clear is that we all work very silly hours…]
@Cameron: anyone in the field will know the gist of your work long before it is formally published, at which point it is old news/a mere formality. ‘Unpublished’ data progresses down a conveyor belt that starts with casual chat, progresses through the formal poster presentation stage and matures in podium talks and invited seminars.
Papers only take longer than a few months to be published because there is an imperative to shoot too high (Nature springs to mind) instead of sending the paper directly to a journal that will accept it straightaway. This, again, is down to the fact that far more biologists are produced than are jobs to house them. If the number of biology PhDs went down, things would get more manageable. I would submit that physicists are in this enviable position already.
Time for my openness rant, although Cameron has done a great job. To me a publication is the formalization of a scientific finding. Somewhere along the way, we’ve made it an avenue to get tenure, amongst other things.
I can only quote Bill Joy
“Wherever you work, most of the smart people are somewhere else”
We benefit from opening up our research. We are more likely to find good collaborators by serendipity. If it wasn’t for the reward system, this entire discussion would be moot.
Of course, I realize that things don’t change overnight (and I am not an academic, so it’s easy for me to spout off). But it’s time for others to join Cameron, Peter Murray-Rust, Jean-Claude, etc and change the game.
To clarify: I’m all for open access at the publication stage (I used to work for BioMed Central, after all). But until you change the reward system, nothing can change. You might be lobbying the cart when really it’s the horse that needs a good talking to.
Jennifer
I think we all agree there. We definitely need to talk to the horse, although perhaps we should be practicing civil disobedience instead
Deepak, that’s really funny; I’ve often wondered what would happen if the entire scientific community went on strike over various vexing issues that they are always complaining about down the pub: the reward system, the pay, the hours. Ain’t never going to happen, but amusing to speculate.
@Jennifer said “confess I can’t parse this sentence at all: I also thought about using a tool like deli.cio.us/www.connotea.org ”
Jennifer, if you don’t know “http://del.icio.us” and social bookmarking I suggest you watch this video: http://www.youtube.com/watch?v=x66lV7GOcNU
http://www.connotea.org is much the same but it works with scientific papers
What I suggested was to use a tool to ‘tag’ you files on your local computer just like del.ico.us can be used to ‘tag’ a web site.
I don’t think the rest of the world would actually notice, Jenny.
And we’d be out of our jobs.
Back again (access to NN is really ropey at the moment or is that just me?). I think Deepak summed up my view quite well. We can do better, we can talk about how far we can/should go; but the people appointing younger scientists to contract and permanent positions need to think beyond simple (and dumb) measures of a person’s worth.
Reward structures do need to (and actually are believe it or not) changing. Though if you look at what people like Pawel Szczeny and others are doing you might consider that civil disobediance…actively trying to move outside the system.
@Deepak: are you calling me a horse? 🙂
Jennifer, I love this comment!
“I was a journal editor for four years recently, and I can soundly attest that many manuscripts don’t pass peer review because they aren’t real. Some are 100% not real, some are part unreal, and some are just plain surreal.”
Thanks, Maxine. The stories I could tell…but of course the editorial office is like a confessional. I would have to be plied with a few drinks, and no names mentioned.
I’ve been thinking a bit about this whole time-stamp issue. I still think it wouldn’t work. People would not be stealing data from an open notebook; they would be stealing knowledge. Knowledge, moreover, which is well known to be stumbled over independently. Ask any journal editor who has ever received three manuscripts on the same topic in the same week. This is not (usually) because two labs stole a third lab’s ideas; it is because the path of least resistance in biology means there are only a few logical, easy extensions building on from a previous paper, and many people are going to be trying it simultaneously. (We can argue about the lack of efficiency of such a strategy, but in some ways repetition is a good thing. Another tangent is why people don’t try for the less obvious experiments – but this is down in part to the broken reward system, again.)
My point is, however, that if my open notebook tells the world that new gene X turns out to be a member of well-known pathway Y, anyone who works on pathway Y can come along and repeat the experiments proving this in a matter of months, where as the screen that led me to make the connection between X and Y may have taken three years. But all the other lab has to do is publish it; no time stamp in the world can prove that this lab saw my notebook and learned about the connection. They could have made the connection themselves, by chance, from their own experiment. Scientific papers don’t need to go into the serendipity of how particular lines of research began (though it would make damn compelling reading if they did).
I would have to be plied with a few drinks
/me goes to the bar.
“No, the bottle. Cheers”.
anyone who works on pathway Y can come along and repeat the experiments proving this in a matter of months
But my point is that you have already got the precedence. Its already out there in your notebook, backed up with the raw data and a solid timestamp. In fact you are better off because the chances are that Lab B are already doing that screen anyway, this being a competitive field.
But what would be even better would be if Lab B saw your result, and got in contact to say; ‘We could confirm this with a screen and write an even better paper’ Both labs could then move on faster and more efficiently, and both you and the other poor postdoc you are competing with get a good paper rather than one of you having to start looking for jobs waiting tables. There is a real potential win-win situation here and many of the people who are pushing the open science agenda expect to get a significant early adopter advantage out of it.
I know this relies on good will and that there are few rewards for being generous. But if getting ahead in science requires being a bastard then I’m not interested. The slighlty disturbing aspect of these kind of conversations is that the underlying assumptiom has to be that you would do this to someone else. After all ‘no-one got ahead by being a nice guy’.
But if we can agree that the science would be better if people were more open (not totally, I’ll accept just more for the moment) then we can get on to how we go about changing the reward system.
As an aside, the UK research councils are starting to push in this direction anyway. Not many people have bothered to read the fine print but anyone on BBSRC or MRC funding is required to make data resulting from that funding available (with ‘reasonable restrictions’ to allow for patenting and publication) for 10 years. There is no exception for unpublished data.
Hi Cameron
OK, got the time-stamp thing now: I didn’t realize you were suggesting that everyone had to time-stamp, that it would be mandatory SOP. Intriguing idea – only the journals could force something like that; they are the ones with the power.
I agree that in a perfect world, your scenario would be brilliant. But I haven’t seen much evidence that in practice, there is enough good will out there to make it work. I hate to sound so cynical, but scientists are just human beings, not some sort of utopian super-creatures – and many humans are territorial, bitchy, selfish, jealous, ambitious and all the other adjectives that makes life so complicated.
Re MRC/BBSRC – interesting, but probably anyone could make a case for the ‘publication’ exception you’ve cited.
I don’t mean so much a mandatory time stamp, but that you would have the first public and verifiable time stamp. Yours in an open notebook, someone elses maybe as a journal submission date but both are verifiable. Now does anyone care about the be notebook date? Well with a paper I’m currently working on (where we were scooped in the literature but had the result out in our notebook first) I’m going to find out. Will be interesting to see.
Re: MRC/BBSRC. People can, but they’ll find that if they do that, they won’t get any more grants in the long term. You currently have to make a data sharing statement in the proposal which is part of the assessment. If later you don’t adhere to that there could be sanctions (although in practice no-one has figured that part out yet, nor monitoring, but they are serious about the need to be able to re-use data).
And as far as perfect worlds go, well I try to live up to those standards (not always successfully) and make the case as best I can. I spent a lot of time as a starting out academic not liking the person I was turning into. Now if I lose out I lose out, but I’m a less stressed and less stressful person as a result. Now I sound like a hippy 🙂
Oh, Jennifer!
“…the reward system, the pay, the hours” – we did this in France in 2003!! Did our pay go up? Were more jobs created for researchers, technicians or postdocs? Is merit no longer decided by so-called “bibliometry”? Did any of you notice outside of France?
No to all of the above.
Now you know what would happen. Precisely nothing useful.
I’m going to give the wiki idea a thought, though; writing in my lab notebook has decreased in inverse proportion to the amount of time I spend on my computer. Perhaps I could have a “wet” and “dry” notebook. My group already has the wiki structure, so it should be easy enough to try out.
@Cameron – Re this: “I don’t mean so much a mandatory time stamp, but that you would have the first public and verifiable time stamp”. Not to be flip, but who cares? In my example above, the Evil Lab who stole my idea that Gene X is involved in Pathway Y publishes their paper to great fanfare in Nature in 2008, and my open notebook shows I made the same connection in 2007. Big deal. There are probably 200 groups working on Pathway Y, and several of them were bound to stumble across the same thing, including Evil Lab. It proves nothing, and wouldn’t stop Evil Lab from doing the same thing again because it could just as easily been a coincidence. But if my notebook had been closed, I might have been the one with the Nature paper and subsequent grant/promotion/Nobel/rock-and-roll lifestyle (delete as appropriate). Does this make sense?
I’m glad you’re a hippy, though! The world needs optimists.
@Heather – my jaw has dropped – this is fantastique! First I’ve heard: do you have any links to the story? How long did you hold out?
@Heather It’s worth talking to the openwetware people about using wikis as notebooks as they have done a fair bit of development work which may be useful to you. I’m not sure whether they’ve rolled out the stuff they’ve been working on for the last few months but it’s going to improve the functionality quite a lot. But do experiment and tell us what you find out because all these experiences are useful in figuring out how to build better systems.
@Jennifer. It all makes sense and I’m not really arguing quite that point. But I think there are two counter arguments 1) with 200 labs against you you’re going to lose most of the time, so having some claim to precedence is better than nothing and 2) from every legal perspective apart from publication you have priority. They can’t patent, they can’t claim moral rights and they can’t claim any technical or copyrights
But at the end of the day I’m not arguing that the world isn’t like that, just that the world needs to change. If not for positive reasons then because if it doesn’t the government is just going to stop paying for it at all. After all, why are they paying £10 billion a year for us to replicate what these other 199 American labs are already doing?
Right, I wasn’t going to do any ‘work’ tonight. Off to read my book.
Cameron…if I hang around you a little bit, maybe I can absorb a little bit of your world view by osmosis. Thanks for being so adamant.
@Cameron … Nah … I reserve quadruped comparisons to the powers that be
@Cameron – Re this “There is a real potential win-win situation here and many of the people who are pushing the open science agenda expect to get a significant early adopter advantage out of it.”
This is something that I hadn’t really thought about before that might have mileage. But I still keep stumbling over the fact that ‘wet’ scientists don’t tend to think like this. What would it take to change?
Armed force and revolution.
What about higher quality chocolate biscuits in the tea room?
Chocolate biscuits ?!
One thing that strikes me in the scooping debate is that there’s a difference between “someone published before me” and “someone stole my ideas”.
As both Cameron and Jennifer point out, in a competitive field it’s more likely that someone is working on the same topic as you – probably using similar methods to you. If they publish before you, is it because they stole your data? Or because they did a better job? Unless you can recognise your data in their work, or speculate that they were “at the conference and saw your poster”, I don’t see how you can know that they scooped you in the “stole your work” sense.
I know many cases where people have worked long and hard on a problem, only to be beaten to publication – it happens all the time. I know of very few – if any – where they have conclusive proof that the competitor benefited from their unpublished work.
I’m all for Cameron’s “idealistic” world view. If you knew what your competitors were working on and could see their progress via their raw data, you could stop wasting your time if they were way ahead of you. Or speed things up, if they were some way behind you. Or avoid having to fix up stupid errors after peer review, by having them pointed out to you (or pointing them out to others). Or best of all – get in touch and work out a course of action beneficial to everyone and the field in question.
In fact the more you think about it, competitive secretive research is a really stupid way to get things done. I think many of us accept it without ever questioning it.
Neil, all good points. I just want to say that in the field of gene discovery in particular, prior knowledge can have a very big impact. It really can take 5 years to work out that a new gene does something, and 5 months to replicate the experiments once you know the connection. For other fields, it’s not so obvious that snooping could be such a tremendous accelerator.
Just to play the gadfly, one argument for competitive secret research is that if multiple independent labs come up with the same thing, the answer could be much more robust. In contrast, if everyone is working in one big community love-in, or if everyone is stepping back because one lab ‘owns’ a line of research, you’re not going to get that messy overlapping Venn diagram of ‘truth’ that can help make understanding stronger.
@ Jenny – “I can soundly attest that many manuscripts don’t pass peer review because they aren’t real. Some are 100% not real, some are part unreal, and some are just plain surreal.” Whatever drugs you’re on, can I have some?
@ Jenny and Cameron – the whole time-stamp thing to establish precedence is analogous to the business of using recorded delivery to mail your intellectual property to your lawyer, who’d keep it on file in case of anyone coming up with the same idea subsequently. Irrespective of the arguments about what constitutes an idea that you can ‘own’, the scheme would break down by force majeure. Group X discovers something which they write down with a time stamp. Group Y makes the same discovery – by the time Group X gets wind of this and has mobilized its army of intellectual property droids, Group Y has published it and has gotten the credit. In the eyes of the world, Group Y has made the discovery and Group X is eating dirt. I know that what I am about to say is controversial, (and that people will say ‘well, he would say that, wouldn’t he?’) but in my view, openness in scientific publication is over-rated.
Sorry, I’m being a lousy hostess. May I make some belated introductions?
“Red flag? Bull. Bull? Red flag.”
(I do agree, Henry.
p.s. The surreal-ness of manuscripts is steeply and inversely correlated with the impact factor of the journal to which they’re submitted. I’m sure it’s much more civilized Up There.)
🙂 The subject of openness in scientific publication is arguably as inflammatory as … well, some other inflammatory subjects I needn’t name.
The surreal-ness of manuscripts is steeply and inversely correlated with the impact factor of the journal to which they’re submitted. I’m sure it’s much more civilized Up There.)
You’d be surprised. Or, on the other hand, perhaps you wouldn’t.
Henry, just stare at the rabbit. Stare at the rabbit.
Sorry, I’m late back in. Have been unable to get on at work (possibly a good thing). Ok:
What would it take to change? A big success story. One where we crack a serious problem in a way that couldn’t have been done except by people sharing data openly. We’re working on it. But chocolate biscuits might also help.
Also I think people need to challenge what is, frankly, unethical behaviour when they see definite examples. Proper attribution lies at the heart of good science, open or not. If we turn a blind eye when people cheat then you can’t be surprised if the system starts to break down.
I’m also not saying that competition is bad necessarily, but that there is a balance to be struck. It can be good as you say, and it is better that similar things get done in different ways to provide a robust data set. When the competition becomes an obsession and millions gets spent on pissing contests then that’s too much competition in my view. At that point I say cut the science budget and build another hospital/school/museum.
And for Henry, sorry yes, a bit of a straw man there perhaps. Just wanted to make the point that there are reasons why other types of priority documents might be important. If we’re not talking about papers but patents then this is critical (at least under US patent law, which is all that really matters, really don’t get me started on that…) and patent lawyers and the people who employ them couldn’t care less about what anyone thinks of them.
But that’s a rather negative example anyway. I think the positive ones are better. At the end of the day, as I think Neil said, there are risks in not being open (you may be scooped), and their are risks in being open (you may be scooped). Its a balance, not a black and white case. For me I think the benefits outweigh the risks and I think for a lot of other people they could as well.
But I’m puzzled by your closing comment. ‘Openess in scientific publication is over-rated’. I mean if we’re going to publish un-reproducible drivel anyway why can’t we at least come up with an automated system so I don’t need to spend so much of my time refereeing the damn stuff.
But seriously, is there any point in publishing if its not useful to someone, i.e. at least open enough to be reproducible. Don’t forget that a significant proprotion of scientific publishers are essentially propped up by massive government subsidies via university subscriptions and page charges. That’s your tax [insert currency of choice] going on providing a glorified set of golf score cards otherwise.
‘As inflammatory as..’ tempting, tempting…
Cameron, that was beautiful. Standing O. Listen, I don’t want to give the impression that I’m anti this whole peace/love-in/open science movement thingie – I’m just a bit cynical. It’s hard to call people on their ethics – so difficult to prove a niggling feeling of bad behavior, and we all know what happens to whistle-blowers even when they’re right. But you’re right we need to talk about this and I’m glad to see it aired here in my humble salon.
About the lab notebook thing, I’ve recently noticed that Google Docs allows users to link to spreadsheets (either created on GDocs or uploaded there) and also insert pictures (which can also be uploaded). That’s practically all I keep in my lab notebook, text describing the experiment, spreadsheets w/ numbers, and pictures of stuff. I’m going to try this out for a while and see how it works out.
More on the labnote book related issues: Addgene Labs is a new non-profit initiative to help biologists manage their information and reagent better, and to help labs run more efficiently. We are adding tools to this system constantly, and one of the tools that will be added soon is a way to help users keep documents, data, logs, and protocols.
This site is a community effort; we are looking for more users and suggestions from early adopters. We’d appreciate your help in evaluating the tools currently available on the site and let us know how we can make this a useful tool for you and your lab.
@Benjie – that looks very interesting indeed – we’ll take a look.
@Nuruddeen – I’m waiting for the beta version of Google Life (the one that goes to work for you so you don’t have to). Seriously, that seems promising as well – I think someone higher up in the thread mentioned this too.
This must be a record for the longest comment thread at NN, and it’s been civil too 🙂
Jenny – I’d like to stare at the rabbit, but I came home last night with a migraine that felt like someone was trying to gouge out my left eye with a sharpened hockey stick. The effect of staring at NN, I think.
Perhaps I should clarify what I mean by ‘openness’. What concerns me as an editor is that the business of dealing with a ms between the time it arrives on one’s (editorial) desk, and the time it is published, should not be subject to the free interference of third parties. On several occasions I’ve had to deal with a high-profile ms, ensuring that the referees have all they need, ensuring that the authors are kept in the loop, while simultaneously trying to fend off the impertinent intrusions of journalists, and the equally impertinent intrusions of funding agencies who want ‘their’ research to come out according to ‘their’ schedule. Justifiable, maybe, as they’re paying for it, but the people I have to deal with are idiotic PR/marketing types who have no idea about the peer review process and the concept that if a paper is worth publishing, it does nobody any good at all to rush things unnecessarily.
As I deal with papers on fossils, my view is generally that as a fossil has been lying in the ground for 500,000,000 years, we can at least do it the courtesy of an extra few days at this end, just to make sure the referees, authors (and me) are happy.
Palaeoanthropology is the worst, as everyone thinks they have a right to comment. I once remember dealing with a very contentious paper in which the authors themselves couldn’t agree about who should be first author, or even on the content of the paper, even after submission. It was a nightmare. Happily I went on leave during the process, leaving a hapless colleague to deal with the ongoing mess.
But that didn’t stop an academic calling me (even when I was halfway across the world) to lobby on behalf of one group of authors or another. I stopped the caller halfway through his exposition, and asked him whether he was one of the authors, or, if not, a referee. When he admitted he was neither, I’m afraid I invited him to do something unprintable.
There is open peer-review, of course, and I have no objection to that in principle – but editors need only collect and collate such open comments when the peer review process is otherwise complete, rather than being forced to dance to someone else’s tune.
Deepak, I have a theory about that.
Having just read Richard’s theory I found myself seized by a testosterone-fuelled, masculine imperative to leave a comment. Some might think this chivalrous. Others might compare it to the urge felt by a dog to decorate a lamp-post. Personally, I blame Charles Darwin. Gee, Officer Krupke, it’s my genes.
@Jennifer: French research directors’ mass resignation (at the time I was a group leader and we got invited to as well) generated a little local-language press and even some brief reference in our host journal, but in the end, the national research budget was bumped up very slightly and then redistributed in nearly the same way. Since most of us are not professional union leaders or even at all attracted to such movements, only the militant fringe seems to remain associated with the original “Sauvons La Recherche” movement. It’s old news now, alas, and still no obvious vision for French science. You can read about it in English here.
@Cameron: Started a little over two years ago in my little spare time. But I’m not really into it enough, so it hasn’t been a lot of use. The OWW lab notebook thing seems very preliminary as yet.
@Henry: my hero. And anyone who quotes from West Side Story has got to be a romantic at heart, kitten torturing habits or no. (I do seem to have a few more comments than I need, though – perhaps I should be chivalrous myself and donate a few to poor Richard?)
It’s amazing how much attention Nature papers get – it is probably difficult to extrapolate this to the rest of the publishing world. It sounds as if you have cause to be angry at the interference your high-profile papers receive, but you cannot entirely blame people for being passionate.
Sorry, the editorial I meant to link was this one. Now I can close the n’th open tab in my browser. 🙂
@Jenny – aw shucks. The kitten business is there just to make me look tough. You know me well enough by now to know that I love all furry animals, and seem to be giving house-room to most of them.
It sounds as if you have cause to be angry at the interference your high-profile papers receive, but you cannot entirely blame people for being passionate.
I think that with any paper, whether high-profile or not, the editors, authors and referees should be left alone until what they have to say is good and ready. Otherwise the science suffers, and that’s really what it’s all about.
Jennifer, I have no idea if this would be useful for you, but speaking as one compulsive documenter to another, I absolutely love the program Evernote
I’ve found it to be very easy to keep a chronological record with all kinds of files and formats, easy to categorize and organize, and very flexible and intuitive to learn and to use.
Re security and backups, I believe you can also password-protect it, and sync it onto other devices.
I am constantly playing around with software, and have tried several note-taking and information filing programs. This one, for me, is the keeper.
Anna, thanks for the tip – yet another nifty device I have not heard of. I was wondering if universities should start running short courses training in basic informatics. It’s probably not a problem restricted to science, but I reckon there would be a lot of interest from people like me, damsels and otherwise.
Heather, it’s a shame your revolution flopped but on the other hand, it’s great that they tried. Thanks for the improved link.
@Heather; OWW has just released some new stuff. Still very preliminary as yet I agree but I think they are heading in the right direction. I think all of the currently available systems have serious limitations but there will be some good stuff in the next 12-18 months.
The things we are working on are also pretty preliminary, which is why I haven’t pushed them here but I think we know what needs to be implemented to build useable systems; its just not a trival thing to build. So watch this space.
Currently I think wikis and/or Googledocs are the best freely available systems.
@Henry: I take your point. I make the assumption that openness is associated with accurate and generous attribution, and perhaps more importantly just general politeness.
In my world view journals act as a filter. The quality and ‘brand’ of the journal is intimately tied up with the quality of that filter. So once you have a paper in your system, what you do with it is entirely your choice, and absolutely should not be influenced by random (or non-random) passers by. Otherwise you’ve lost control of your brand.
Nature is what it is because of the quality and perception of quality of peer review. The idea that people even think they can influence the journal by putting pressure on the editors has the potential to diminish that.
Absolutely. If only authors realized that harassing the editor would only have the opposite effect (if, of course, their objectivity could be influenced by annoyance!).
GoogleDocs sounds like a good departure point for me – I need to work my way up gently.
Sorry to join this late.
One fact of life in “open science” discussions is that there are going to be people who think it doesn’t make sense in their (‘very competitive’) field to be open about their work. However, there are many, many scientists whose principle problem isn’t being scooped — it’s that no one notices their work. This is especially true among younger scientists still making a name for themselves or folks in smaller fields.
I think there is already significant incentive for young scientists to publicize what they are doing as openly and early as possible. This open group will either be scooped out of existence, or will be more successful thanks to all the unintended benefits of making your work accessible early. We really won’t know which it is until we run the experiment, but you can probably guess where I lie on this one.
I think the points about time-stamping above are interesting and might be right, but the first group of people who open up will do it totally unprotected. If we need elaborate infrastructure and a change in the scientific reward structure before people open up then we’re dead in the water.
At OWW we’re trying to create a community that values openness as early as possible in the research process. We support this community by providing simple web tools to make the process of sharing info as easy as possible. The OWW notebook is a new tool that tries to follow that model — it’s very simple and it’s based off wiki templates that have been battle-tested by community members previously keeping open notebooks on OWW. Might call it preliminary, but I’d call it ‘light-weight’ 😉 Give it a try (it works especially well for coordinating a group project), or just join the community and share what you’re up to.
Thanks, Jason. That’s really interesting. Do you have any small fields in mind, as specific examples, in biomedical science? I’d be intrigued to hear which you think fall into this category.
Well, the easiest field to speak about is the one I’m in – Synthetic Biology. This field has grown quite a lot in a short period of time (1st conference was 4 years ago), and I think it has benefitted tremendously from a commitment to early, open exchange. Both on the web (syntheticbiology.org), and through things like the iGEM competition. As a result of iGEM most of the faculty in the field (and many students) get together once a year to talk about projects that are run largely in the open.
There are also opportunities for community (inter-lab) projects, for instance in the area of standard-setting. As an example, I recently sent an email out to the synthbio community mailing list asking if anyone would be interested in running my standard constructs in their lab so I could get data on variability across groups. I got back 4 replies, mailed strains, and data started popping up on the wiki within a week. Two of the folks running the experiments were people I’ve only interacted with online. I think my paper will be better for these results, and I doubt I would have been able to leverage a community like this if I hadn’t been open about the work.
Another example on OWW is the Mimulus community – a model plant I hadn’t heard of before they popped up on OWW. This is really just a community that needed a low barrier way to communicate and collaborate since labs are scattered all over the country.
I think it is early days for a lot of this stuff, but from the perspective of a PhD student being open early is a major win. A very small fraction of PhD projects are even “scoop-worthy”, and a very large fraction spend time down dead-ends that could have been avoided by the right person noticing a mistake. Openness wins in this case, IMO.
Great story. Many thanks for that. {Note to self: found new field.}
The irony, of course, is that data-sharing, if it’s as win/win as you say, will inevitably make a small field larger and more prestigious – and probably more cutthroat. I’m thinking of C. elegans – back in the day, people used to talk about it in similar terms, but now, they say it’s just as bad as any other.
What do you all think this observation about data-sharing and open science? I have seen a lot of motivations for data-sharing and open science that stem from the engineering field (I am a trained CS person), but I think there is a fundamental difference here. It’s like the “zero-sum” observation in econimics: in engineering there is not a zero-sum; engineering is kind of like an art, there are many ways to the same solution and each can be considered a novel engineering feet, depends on the trade-offs and parameters. But in science, there is often a zero-sum. If you are trying to discover something, once it’s discovered, no one can discover it again. So there is a dis-incentative built-in for not sharing, because of the zero-sum.
Come on Jenny, what’s your secret? The Damsel-in-distress theory doesn’t adequately explain nearly 100 comments.
But it’s good to see some serious discussion and interesting comments going on here. So keep those comment-generating posts coming thick and fast…
Benjie
That’s a bit of a limiting view of science. If it was all a zero sum game, then we’d not get too far. Especially in biology where there’s always something more to discover 🙂
Wow – this is a long thread indeed! Very encouraging that people are eager to engage in conversation about science and openness.
Neil, Cameron and Deepak have accurately represented my thoughts on the matter.
It really comes down to our values and objectives. If your livelihood depends upon securing patents then you should not expose your laboratory notebook to the public. However, if you are interested in finding collaborators or becoming more visible (as Jason pointed out) then exposing your notebook to some extent may be an effective strategy, in addition to traditional publication, conferences, etc.
For my research group, switching to Open Notebook Science has been extremely beneficial for finding some great collaborators (and friends). We’re working on making anti-malarial compounds so I would be quite happy if someone branched off from our ongoing results to do something useful or even point out another interpretation or error.
If they do use the data I would like for them to respect the CC-BY license on the wiki and use proper attribution. If they don’t then it will make for an interesting topic for me to blog about:) It is going to be much easier to describe who-did-what-when by linking to documents with third party timestamps rather than claim that I had that idea in a proposal that was never made public and shown to my closest competitors under the cover of anonymity.
But beyond this, some of us (Cameron and Gus Rosania especially) think that science can be done much more efficiently under Open Notebook conditions. We’re testing this out and are reporting what we find. So far so good.
I also think that this is how we’re going to transition to a whole new level of automation in the scientific process – but that is a separate issue.
By the way, a general purpose wiki (open or closed) makes a pretty good lab notebook. I use Wikispaces – it is free if you make it public – $5/month for keeping private.
Five more comments (um, four now) and Jenny becomes the first NN blogger to reach her century!
Great post Jean-Claude.
I posted this yesterday here at NN to hopefully, generate further interest in your talk yesterday and the panel discussion. Was great to watch the web cast.
(Only 3 comments to go now…)
Grant pushes away to the onside, and they’re going to snatch a quick single.
Steel just makes it for another quick single before lunch. Umpire Brown takes note.
The crowd looks on in antici……………pation.
He shoots! He scores!
Some people are on the pitch… They think it’s all over………It is now
The crowd goes crazy and Gee is still running round the pitch like a mad scientist (he forgot his coat) after scoring the point that makes Rohn a NN record breaker.
Tea anyone?
… puff … pant … puff… pant … puff… where did I put my coat?
Tea? Tea? I could murder a Babycham.
And the scenes here at Lords are amazing. Gee is running along the wicket, he’s going to straddle the stumps, and –
oh dear, he didn’t quite get his leg over.
Story of my life, Brian.
{blush}
Don’t you people have lives?
Thanks, all.
xxxx
I have a life, Jennifer. It’s called Nature Network.
“The Life Of Brian/Henry and Co.” (ze. Film) be too much..
Yes but no but yes but I’m
A-fraid
bq. “I have a life, Jennifer. It’s called Nature Network.”
That’s so going on our marketing materials.
It’ll cost ya.
Always happy to provide a useful service.
You might want to check out BioKM, it is an online laboratory management tool that helps you organize all of your lab materials and research results. BioKM enables users to manage all of their materials, protocols and data in one place using a project management-like system. Users are able to upload their experiments and results in a secure environment – perfect for a ‘compulsive documenter’ like yourself! You can find out more about BioKM by visiting http://www.biodata.com.