Over at Myrmecos, Alex Wild was musing about blogging about peer-reviewed research.
One of his commenters, Psi WaveFunction, commented
I don’t get the whole peer review worship going on in the general public as well as academia. It’s almost as if people in the whole WANT to keep scientific knowledge locked up accessible to few and practiced by even fewer.
This got me wondering – why do we consider peer reviewed research to be important? I’d actually suggest that peer review is of secondary importance, so I’ll discuss that later (if I remember). So, why do we need a scientific literature? Well, here are my half-formed thoughts.
h3. Why do we need a scientific literature?
Science is accumulated knowledge and understanding. We have to store it somehow, and this is what the scientific literature is. All those experiments, models, observations, speculations, and massively boring introductions are a part of this. If we are to refer to science as a body of knowledge, we need to be able to point to it (metaphorically!).
Does there need to be a “formal” literature?
I think so. With a formal literature, we can refer and cite work in the literature with at least some confidence that it is trustworthy. Without a recognizable formal literature, it is more difficult to separate out science from pseudo-science, popular science accounts1 and most mindless speculation.
Aside from any necessity, I think some form of formal literature is inevitable sociologically: as a group, scientists want to communicate with each other, which can be done more effectively by allowing a jargon that defines obscure concepts with precision. It just makes it easier for scientists to communicate with each other.
What does a formal literature look like?
The formal record has to have a feeling of permanence: we should be able to read it many years later. I guess taxonomy is the epitome of this, where one regularly sees reference to works from the 18th century, by The Scientist Known Only As L. But this holds for other work too: I have non-trivially cited work from the 1940s.
Of course, the work also have to be available to scientists: there is no point in revising the family Pyroglyphidae if you’re going to keep the only copy of your manuscript in a dusty attic.
This permanence and availability implies that a certain amount of organisation is needed. The work needs to be produced in enough copies that it can be distributed widely. It also has to be archived, so that it is can be found and read, which means librarians have to be involved. So, we need an infrastructure to manage it.
How do we decide what should be part of this literature?
Obviously the literature has to fit with the (current) norms of science, which means that someone has to check it. This means that there has to be some level of peer review, even if not the form it is now. One could imagine, for example, a literature that consisted solely of examine theses: even professors would have to write a thesis, and be grilled on it by a colleague. Preferably in public, garnished with salt and herbs. There are, presumably, other ways to filter the literature: none will be perfect and the one we have now has accumulated rather than been designed, so there is no guarantee that it is the best way.
Does this literature have to be inaccessible to non-scientists?
No, and sort-of yes. No, in that it should be available to anyone wanting to read it. At the moment a member of the general public has to do some work to read it: order from the local library, or email the authors to ask for a copy (hint: if you email L., make sure you specify that he should reply during the day), but there is nothing intrinsic in the literature that makes it unavailable.
A lot of the literature is still inaccessable in practice, because of the way it is written. This is a thorny issue. Ideally, it should be understandable by anyone, and not bogged down in jargon. In practice, of course, a lot of the literature is only really comprehendible by other experts. I’m not sure to what extent this should be condemned. Any communication, I think, is exclusionary. This may be as simple as excluding people who don’t understand the language, or there are explicit attacks on a group of readers. But it can also be subtler, perhaps the tone and style will be off-putting for some readers so they decide not to bother following. Or just the subject matter is uninteresting. The intent may not be to exclude, but that is the effect. The scientific literature does this, again not deliberately to exclude, but simply because its target audience is other scientists, so it is written using scientific terminology, and with an assumption of a decent background knowledge: we expect the reader to understand “t=3.43, 9 d.f., p=0.0075”. This is necessary if we are to communicate with each other effectively. Of course, this means that part of learning one’s trade as a scientist is learning to understand the language. This creates a barrier to the general public, and there is an obvious trade-off between communicating to a wider audience or efficiently to a smaller number of specialists.
Should the formal literature only cite other parts of the formal literature?
For me, it is obvious that the formal literature has to be able to cite other literature, whether it be newspapers, blogs, or even pers. comm. For some of this literature, it will be unclear whether it is part of the formal literature or not. I think this is inevitable: I’m not sure our definitions of the formal literature has to be so proscriptive that everything is either in or not in.
Do we need peer review?
I think at some level, yes. If we are to have a literature with rules (whether formal themselves, or just accepted social conventions), then entry into that literature has to be assessed, and this is what peer review does.
This does not necessarily mean that the current system of peer reviewed journals is the best, or that the rules that are used (and abused?) are ideal: that’s a slightly different topic. My point here is that if we have a body of literature that only includes some work, then it needs gatekeepers to make sure that only the right work is included (modulo the usual vaguaries).
Should we privilege the formal literature over other scientific writings?
I think to some extent, yes: I suspect everybody does. The controversy is (I think) over how much. We certainly shouldn’t view everything in the formal literature as being correct: this is not a theological canon. However, we should expect it to have some features that we may automatically expect to see in other writings: it should have be about science, and have been prepared with care and precision. How much more we expect of it depends on the way the scientific culture has evolved: we expect a certain level of caution and neutrality of tone because that it the way it has developed, rather than as a necessity. of course, the purpose of these mores is to give clarity, but it is not obvious to me that this is the only way of achieving it.
From this, I suspect that the commenter’s frustration with the literature is not frustration with the necessity of it, but rather with some of the “optional extras”, due to the way it has evolved. I’m intrigued by the idea that there could be a minimum specification for a formal literature: this would give us some idea about how far we could push innovation. The development of the web changes the publishing landscape, and is leading to innovations in how the formal literature is produced. So can we seem the limits of this innovation? If we can, we might see that we are actually freer to experiment than we think.
1 This is not to say that there is anything wrong with popular science, but inevitable it will skip over some details.
This is a very thoughtful post, Bob. I agree globally with all of your conclusions, although I would say that natural selection operates well in the arena of the diffusion of scientific information.
Formal literature survives because it is a system based on establishing and maintaining trust. I would have read a paper in certain journals more critically perhaps than in others, where I can bypass some of the necessary skepticism as I am trusting my peers to do some of this work for me. In my own field, I will be more critical than in auxilliary subjects.
I think a lot of people get messed up over ‘peer review’.
Surely at its most basic peer review says that (a) something has been done in the right way using the right methods and (b) it hasn’t already been reported.
I can’t see how you can throw that away without completely foxing the garbage bag.
Great post, Bob. It occurred to me while reading is that the exclusivity/accessibility/ seriousness of scientific literature ought to be rated by some kind of Beaufort Scale. Such a scale already exists for general reading level, but I was thinking of one specially designed for science, you know, like Force 6, intelligible by most biologists, chimney-pots dislodged.
Should I be worried that people are agreeing with me?
Good point, Heather, about trust. I guess that’s one of the important reasons for having a recognizable literature. I wish I’d thought of it. Except the post would have been even longer….
Tangentially related to Heather’s point about trust is transparency, especially in medical literature. Most journals now are fairly thorough about publishing potential conflicts of interest for the authors. However, I think peer-reviewed journals would (I hope) be more thorough in identifying these types of issues.
Lovely post, Bob. You raise some really interesting questions:
I think talking about accessibility for non-scientists here is a bit unrealistic. The scientific literature (within a discipline) should be accessible to someone who has a (Bachelor’s/Master’s) university level education.
I think your t-test example is missing a little something
Hopefully, the author mentioned somewhere in the methods that treatment groups were compared using a t-test – this sets the context to allow easier understanding of this “jargon” when it appears in the results. At least an interested observer can then go an check out what a t-test is in a stats book1.
We need extensive training to understand the background and basic concepts of our discipline, we can’t really expect those with a different educational background (for non-scientists, this could be a lack of science education from the age of 16) to be up to speed with every technical advance we offer in our manuscripts. Unless you want to make your Introduction even longer. Say, the length of a 4 year undergraduate course…
1 or on Wukupedia, New Zealand’s favourite online reference source.
Re. accessibility to non-scientists, I think there is a case for making a lay abstract available that explains briefly what was done and why it is significant. That way non-specialists can at least understand a bit of the article. It might also make it easier for scientists in other disciplines to understand.
I just read an interesting (long) post by Michael Nielsen on how scientific publishing is being disrupted.
Great post Bob. In large, I think I also agree with you major points.
In terms of accessibility of literature, many open access journals seem to be bridging that gap between non-scientists being able to access a journal article of their choosing, however, I think that scientific language is critical. Frank makes a good point though. Perhaps a lay abstract is a good way to inform the non-scientist reader. After all, we should all be able to (or have done) this and it would be a great way to communicate with those interested in your research but not necessarily in your field (or even scientists).
There’s nothing in here for me to disagree with… who are you and what have you done with Bob?!
Bugger. If even Bill agrees, I must be doing something wrong…
Caryn’s points about OA are, I think, critical. just because something is available to be read doesn’t mean it is accessible: it may be impossible to understand. I’m not too worried about that: if the primary audience for the scientific literature is other scientists, then that’s who it should be written for. I guess where that argument runs into problems is when the criteria for becoming a scientist are set so high that it’s difficult to enter the profession, or to be able to follow it.
@Caryn – Perhaps a lay abstract is a good way to inform the non-scientist reader.
A great idea. Why doesn’t Nature show the way? They could ask for the abstract only once the paper is accepted (in principle) and make publication conditional on receipt of an acceptable lay summary. That would be good incentive, I think!
We do ask for a lay summary, and authors provide it in their cover letters. From the Nature Guide to Authors:
5. Cover Letter
Submissions should be accompanied by a brief covering letter from the corresponding author. This letter should contain two (100-word or shorter) summaries: a concise paragraph to the editor indicating the scientific grounds why the paper should be considered for a topical, interdisciplinary journal rather than for a single-discipline or archival journal; and a separate, 100-word summary of the paper’s appeal to a popular (non-scientific) audience.
These can be used by our various summary writers and press-release writers if they want to – sometimes they are but not always.
The big advantage, in my view and I think that of my colleagues, is that asking an author to provide these summaries at submission rather than upon acceptance really makes them think about “why Nature and not a specialist journal” and “why would the general public find my results interesting” while they are preparing the paper itself – resulting in crisper prose and more focused argument.
Maxine, that’s interesting – I find it much easier to write summaries after the paper (or grant) has been written, and the PIs I’ve worked with on manuscript and grant applications have all done it the same way. Mind you I’ve never been involved in preparing a submission to Nature. Do you get much author feedback on the summaries?
Yes, they like to write them. I am pretty sure they write them after the paper has been written, on the whole, but our idea in asking authors to submit them with the manuscript is to get them to really focus on why their paper would be suitable for a journal that rejects virtually all submissions, most without peer-review. So part of the idea is to help the scientists to get over that hurdle – or perhaps to realise that there isn’t a sufficiently startling conclusion in that particular piece of work, and saving themselves some time/pain.
I honestly thought Stephen was being tongue-in-cheek… but I realize that he just doesn’t read Nature.
😉
You’d probably only realise this is our policy if you are (1) a
serialregular submitter or (2) a bit of a nerd who scours through journal Guides to Authors and policy pages. We’ve announced it and mentioned it in Editorials a few times over the years.Are these summaries the “Editor’s summary” that a lot of articles have?
No I just look at the pictures, Richard… 😉
I don’t fall into either of Maxine’s categories, alas. While I take her point that Nature clearly do ask for a summary of each paper’s popular appeal, this doesn’t seem to be made available for every paper that is published. As I am flicking through the display items, I cannot fail sometimes to notice the abstracts and I often find it difficult to appreciate why a particular piece of work has appeared there, rather than in a more specialised journal. Of course this applies mostly to papers outside the life sciences.
Hi Stephen – we don’t publish the author’s summary as-is – but we do publish an editor’s summary (about 100-200 words) of every paper in Nature. A selection of summaries are in the printed issue immediately after the table of contents; the summaries for the entire set of papers in an issue is on the journal’s home web page. Also, each paper is linked online to its editor’s summary.
In addition to this, each paper has a very short summary consisting of a sentence or two, which is on the online table of contents (TOC) listing and the TOC alert you can get by email or RSS.
The Nature staff editorial summarisers and the press-release writers use the authors’ summaries as the basis for their summaries (if that isn’t too confusingly put, sorry). But the paper has gone through so many rounds of revision by the time it is published, that the orginial summary is often inappropriate in too many ways. Whether we should at the point of acceptance ask the author to revise it is an interesting question – giving that we are already providing readers with editor’s summaries. What do you and others think?
I should also note that some of the Nature journals make their press-release summaries public – you can read those via the journals’ websites. Unfortunately we haven’t been able to do this for Nature yet for technical/resource reasons, but it is on the list and will happen – I think as the year is 2009 I can confidently say, this century 😉
Pictures – not such a joke, actually! Chemistry journals lead the way in this. If you look at the TOC of Nature Chemistry or Nature Chemical Biology, for example, each (or most?) articles have a small version of the crucial image (usually a chemical structure) embedded within the TOC listing. Again, it would be great to do that for Nature and other Nature journals. It is just a question of how much time and resources there are to implement all these things, and what priority each should have.
Views on all this are most welcome, and I always take to our editorial discussions the useful feedback and suggestions we get via Nature Network.
Slight correction- our summary writers sometimes, but not always, use the authors’ summaries for their own summaries. It depends on various factors – including how good the authors’ summary is or isn’t; whether there is richer information in the editorial deliberations that have accummulated on the file by the time of acceptance; the attitiude of the summary-writer (some prefer to start from scratch), etc.
Heh, I’m not sure I can be categorized that narrowly either!
Aside: I was really chuffed when my paper was published in Nature Structural Biology (as it was, then) along with a picture in the contents… of a fluorescent cell (from the paper). Cell biologists so totally rock.
@Maxine -Ah, my remarks referred only to the printed version; I tend not to use the online version for browsing. I’m losing the argument here (no great surprise), but it seems a shame not to have all the editor’s summaries in the printed version. Surely that would only add one more page?
Anyway, I have another gripe that will likely erupt in these here parts someday soon…
Oh, exciting, Stephen, can’t wait to read that!
On the summaries – the print edition’s design has a double-page spread for the summaries, so we’re curently limited by that. We are definitely looking at ways to get all the summaries into the print edition – they are very popular with readers. (The Research Highlights are also very popular – this is another double page spread of summaries, but to papers in other journals published in that week, selected and written by the editors, edited by Brendan Maher – who Nature Network readers probably know as one of the editors who moderates the Nature Opinion forum here.)
Terrific discussion Bob. I particularly liked what you had to say about citing other forms of literature. I just linked to this post here.
What is the importance of peer reviewed scholarly journals to the study and application of public health?