Busy

Were I to summarize the essence of my job at your favourite weekly professional science journal beginning with N in a single word, it would be ‘making decisions’. (Well, that’s two words, actually, but, hey, I’m a palaeontologist, so sue me). Almost always my decision is ‘no’.

An Editor at Your Favourite Weekly Professional Science Journal Beginning With N, yesterday.

I reject at least four in every five manuscripts straight off the bat, before review; most of the rest perish in review (Note added later: some people are shocked to hear this, perhaps assuming that we send out for review everything we receive). In the end, barely one in twenty new submissions makes it through to publication.

One reason for maintaining a very high rejection rate is to ensure that the quality of the material we publish is always high (which I maintain it is, even when one accounts for the fact that editors are only human and prone to err). This creates a feedback – if the quality is high, and is perceived to be high, people will tend to submit their best stuff to us. Were we to loosen our belts, lowering the bar (and increasing pagination) the quality would lessen, people would send us their just-about-okay stuff as well as their best stuff, and, well, it would be the end of civilization as we know it. In a crowded market, it’s always been a good strategy to offer a high-end product.

The operative word here is crowded. For I have recently discovered yet another reason why editors at your favourite weekly professional science magazine beginning with N, and for all I know more ordinary journals, should be even more discerning.

That reason is the increasingly limited time and patience of referees.

I have been at your favourite weekly etcetera etcetera and so on and so forth for almost a quarter of a century. Back in the day, we used to get reports by mailing (remember that?) manuscripts to potential referees without even asking them first, and be reasonably sure that we’d get a result. These days we send email requests, and, in general, referees are happy to oblige. This is, in fact, remarkable, given that the referees do not get paid for this; they accrue no academic credit for it; and it takes the bulk of a working day to referee a manuscript properly. In many cases, the referee will see the same manuscript in several iterations before a final decision is made.

The system is maintained because today’s referee is tomorrow’s author, and authors will, naturally and reasonably, expect their manuscripts to be treated with the same care, courtesy and expedition as they would accord anyone else’s manuscripts. To be sure, the system is open to abuse by what we in the evolutionary game-theory biz call ‘defectors’, but on the whole it’s what we in the same biz call an ‘evolutionarily stable strategy’.

During that time I’ve been an editor at your favourite weekly panjandrum I have seen a secular growth in the number of manuscripts, and the number of journals in which they might be published. In the mid-1990s, for example, I headed up Nature‘s popular science writing arm (a very distant ancestor of what eventually became this), while at the same time handling all manuscripts in organismal biology as a part-time job. Today, the manuscript load in organismal biology has more than tripled, and it takes at least two of us, working full time, to keep up with the Red Queen, who in our case is anything but hypothetical (NB – this is an ecological in-joke). Year on year, I have ever tougher targets: the percentage of manuscripts I can afford to accept gets smaller and smaller. Over the same period, NPG has spun off a large number of top-tier specialty research journals that didn’t exist when I joined, and other publishers have done the same.

More manuscripts!

More journals!

Whee Doggies!

IMG_4527

Article-level metrics? Post-publication peer-review? Wake me up when something happens would you? Zzzzzz.

The number of editors has increased over the same period, but hasn’t kept up with the volume of manuscripts. There is a good reason for this – more editors would handle more papers and increase the demand on page space in the journal, which, as I have said, benefits from limiting its pagination – Nature‘s pagination hasn’t increased markedly in a generation: where new pages are added they tend to be for news and magazine-style features rather than primary research (and – cynical aside – publishing more research papers does dreadful things to one’s journal impact factor).

But I digress.

I suspect that whereas and notwithstanding the number of potential referees has also increased over the same period, as with editors, it hasn’t kept pace with the increased numbers of manuscripts. For referees are authors, and because research excellence is measured in coffee spoons publications, scientists (I suspect) churn out more manuscripts than they used to. This increases demands on the space of top-tier journals which (for the reasons I have explained, hey, keep up there at the back) reject more, creating a market for specialty spin-offs. Frank can no doubt explain the impact of this on research libraries and their budgets – but what does it mean for authors and referees? It seems pretty obvious that as authors are submitting more, they are also being asked to review more. Not only that, they are being asked to serve on editorial boards of journals more, if not to serve as editors, in which they have the task of finding suitable people in a limited pool to assess an ever-increasing tide of manuscripts. All uncredited. All unpaid. None of which accrues brownie points.

In the journals the scientists come and go, towards absurdam, reductio.

Now, where was I?

Ah, yes. Now, I’m not sure if it’s a blip, but over the past couple of months I have seen an increase in the incidences of referees declining to review manuscripts because they are too busy. They will, increasingly, cite the reasons for being too busy – serving on editorial boards, being asked to review too many manuscripts and so on. One referee complained – in not so many words – that the authors of a particularly recalcitrant paper weren’t taking his comments seriously, which, given the time he was spending on it, seemed unfair. This hit home, given that this particular referee has been reviewing manuscripts for your favourite weekly oompa-loompa for a long time, and is valued for his informative and conscientious reports – reports we use to make informed decisions.

What, then, is to be done?

I know what I would do – and what I am already doing, and always have done, only this time more consciously than before. Given that we at your favourite weekly brouhaha absolutely depend on the goodwill of referees, my task will be to use their services even more sparingly than I do already, and reject manuscripts even more ferociously at the first cut. If the job of an editor is to make decisions, what is the nature of such decisions? In short – that there is (or is not) an ironclad editorial case for publishing a manuscript, irrespective of its technical merits.

It is the referee’s job to judge those technical merits, once an editorial decision has been made. There is a tendency for editors, in a state of indecision, to float manuscripts off to referees in the hope that these (unpaid) referees will do their (paid) editorial job for them. This was and is always bad practice (at least in my personal opinion), but matters now are more acute than ever, given that the time of referees is so valuable. If we, as editors, send manuscripts out to review if – and only if – we think there’s a pretty good chance they’ll be good enough – then the job of referees will be easier and more pleasurable (long ago, I remember a scientist telling me he enjoyed reviewing manuscripts from your favourite poobah because, in his words, “you don’t send me crap”). Referees will know that, whatever else might happen, a manuscript from your favourite WPSMBWN will be pretty decent, and make time to review it.

Having kept our referees happy, what then? Editors will make more decisions based on their own recognizance or after consultation with colleagues, and reject more manuscripts without review. This will put more demands on the instincts, experience and judgment of editors, who, believe it or not, are only human.

I can’t see what’s wrong with this, and indeed cannot see any other future, so long as authors, referees (and, let it not be forgotten, tenure committees)  value the imprimatur of a top-shelf top-tier publication. It’s possible that schemes such as post-publication peer review might offer some relief, but – and it’s only a guess – in some perhaps rather specific subject areas, rather than for the scientific endeavour more generally.

Phew. And after all that, a survey showed that eight out of ten referees preferred their cats.

We can haz paper in Nature pleez?

About cromercrox

Cromercrox is a recovering palaeontologist, author and editor who lists his recreations as writing, beachcombing, playing hard rock organ, supporting Norwich City FC and falling asleep.
This entry was posted in Research, Science Is Vital, Writing & Reading and tagged , , , , , . Bookmark the permalink.

21 Responses to Busy

  1. Jeff Crook says:

    Have you ever done a study of the population of scientists compared to the actual number of journals in which they might publish? Although there seem to more journals than ever before, it could be that they are woefully insufficent for the habitat needs of your scientist population. They may be reaching a dangerous saturation point at which the habitat itself collapses, resulting in mass starvation and whole populations of scientists moving in search of better grazing. If so, I suspect we’ll soon see an explosion in the population of mad scientists conducting un-peer-reviewed experiments resulting in the inevitable monsters and the movie executives who feed on them. Things could get ugly, yet entertaining.

  2. Steve Caplan says:

    Henry,

    Just a couple comments:
    1) I don’t agree that reviewers “get nothing out of reviewing” just because there is no monetary gain. First, researchers do add their reviewing work (in general) to their CVs–particularly editorial boards. Second, there is a motivating factor to be “in the know” about relevant work done in the field. Not to steal or copy it, but to know what’s going on. And the critical reading of like manuscripts often does inspire ideas for one’s own work.

    2) I’m not faulting your journal (in which I’ve not published, but have published in the Cell Biology “spinoff” as you call it), but everything that you describe is highly specific and relevant to your journal and only a few more. For the most part, I am very much against paid editors who are not doing actual science making decisions about what to send out to review. This has been a disastrous thing for several journals that I could name, and has led to a plethora of problems. So while it might work for N., this is NOT a good model for other journals.

    • cromercrox says:

      Yes, I guess there is some benefit to refereeing, as you say, in terms of inside knowledge – and, I suspect, people are in some cases flattered to be asked. However, this has to be balanced against the time commitment of refereeing a paper and the pressures of work that yield more immediate benefits.

      Without naming names, might you be more specific about why you think professional editors are in general a bad thing?

      • Steve Caplan says:

        “Without naming names, might you be more specific about why you think professional editors are in general a bad thing?”

        From my experience with several of these very high tier journals that have professional editors, in my field the tendency to send out to review seems to depend entirely upon the intuition of whether this person thinks the topic is “hot”. On the one hand, this is fine, because the assumption is that the science is rigorous, and if not, then the reviewers will disqualify. The problem arises in what qualifications the professional editor has in deciding whether my manuscript is going to be “hot” or not.

        If we go by impact and citations afterwards, if your favorite journal N. has an impact factor of approximately 35, and my paper is deemed “not hot enough”, but later grabs 100+ citations in a more specialized journal (that is similarly rigorously peer reviewed)- then (and this has happened to me a number of times- no journal names noted) I would contend that such a professional editor is not capable of distinguishing between the “hot” and “warm” papers.

        Now if you think that in my field the professional editors are also discerning on the basis of the rigorousness of the science–I disagree with that altogether. It may be the case in your field, but not in mine.

        • cromercrox says:

          I think this ‘what’s hot’ business is a caricature.

          Professional editors look for what’s conceptually novel. Of course, novelty will correlate with ‘hot’. As I wrote below in response to AJ Cann, professional editors have a very broad knowledge, whereas scientists will know smaller areas in greater depth. This gives professional editors a context – and an experience – that scientists don’t generally have. This is why editors handle mss outside what a scientist would consider their immediate field. It doesn’t take deep technical knowledge to decide whether a mss is editorially suitable or not.

          Now, when a journal rejects a ms before or after review, based on novelty, or advance over previous work (but not ‘hotness’), and the ms gets more citations in a specialist journal elsewhere, this could mean any number of things.

          1) It could mean that the editor at the general journal misjudged it. Editors are only human. That’s why we have appeals procedures (a misconception I come across very often is that Nature doesn’t allow appeals).

          2) It could also mean that the journal in which the ms was published was more appropriate than the general journal, and more visible to the right audience.

          3) It could mean that the ms was substandard when it was submitted to the general journal, rejected, and then published in a speciality journal after benefiting from the editorial time and advice and the refereeing lavished on it while it was at the first journal. So when authors in such a situation throw brickbats at the general journal, it’s deeply unfair – especially as editors are bound by confidentiality not to divulge the identities of mss that have passed across their desks and which have been rejected.

          • Steve Caplan says:

            I will give you “partial credit” on that response–

            1) I agree that editors are human and sometimes subject to bias from “big names” and “big institutions”. The problem is that sometimes they are “too human”…

            2) If highly cited in a specialty journal, it would be even more highly cited in a “general journal”–because the specialty people will see it wherever it’s published, but he general people might not.

            3) You talk about the authors “benefiting from the first review at the general journal where the ms was rejected”–but I am talking about rejection by the professional editor–so there is no critique, just a note saying “Thank you for your submission, but the paper was not not sufficiently mind-blowing for publishing by our establishment.

            So I respectfully disagree that #2 and #3 are valid reasons why a paper can garner multiple citations from a “specialist” journal despite being rejected editorially from a specialist journal.

  3. Mike says:

    Henry, I have to say I appreciate your candour and approach. As a reviewer. It takes me at least a day to properly review a good MS. Possibly longer for a bad one. I’d suggest that anyone who takes less time than this in the field of theoretical population biology ain’t doing the job properly. This is partly because mathematical/simulation models can generally be replicated and therefore verified on a computer pretty quickly, assuming the methods are all provided. There’s also a stack of relevant literature out there from the last 50 years that authors often forget about.

    As a submissive submitting author, your attitude fills me with peril.

    As far as cats are concerned, I’m number 9 or 10. Give me that gorgeous (quadra) leggy blonde any day. Get down, Shep.

  4. cromercrox says:

    Just seen this link http://scienceoftheinvisible.blogspot.com/2011/02/this-isnt-peer-review.html
    and have updated the post accordingly. Some people affect shock to learn that we reject most of what we receive without review. Where have they been?

  5. Cath@VWXYNot? says:

    How did you manage to write this whole post without using a “pier review” pun?

  6. AJ Cann says:

    Not at all shocked. If you’re telling us peer review is broken, I’d agree. If you’re telling us contributions to nature are peer reviewed, I’d disagree.

  7. Cromercrox says:

    @Cath: it never occurred to me, honest.

    @AJCann: I’m not sure I understand where you’re coming from. I don’t think the peer-review system is broken, so we’re going to have to disagree. But every research paper and review article that’s published in Nature is peer reviewed. Thoroughly. Usually at least twice.

  8. AJ Cann says:

    By definition choosing to reject the majority of papers which are submitted is not peer review, it is an editorial process. Saying that every research paper and review article that’s published in Nature is peer reviewed is like saying that every Nobel Prize winner that has published in Nature has published in Nature – it is a circular argument. Why not send all manuscripts received out to peer review (by peers, not by Nature’s employees)? Because the peer review system is broken. Or if you don’t like that idea, there’s always post-publication peer review ;-)

  9. Pingback: Tweets that mention Busy | The End Of The Pier Show -- Topsy.com

  10. cromercrox says:

    By definition choosing to reject the majority of papers which are submitted is not peer review, it is an editorial process

    Well, of course it’s an editorial process. I don’t think I’ve said otherwise.

    Why not send all mss received for peer review?

    Because, as above, we have editorial criteria to which we adhere, and which are publicly available. Mss submitted to Nature must pass these before we send them to be reviewed externally. Referees would soon get fed up if we sent them everything we receive – not because peer-review is ‘broken’ – but because they would say – rightly – that there should be people like me to screen the submissions first.

    This raises two questions. First, whether one should employ any editorial criteria at all, and, second, whether professional editors are suitable for this job. Any magazine editor has the perfect right to define editorial criteria in any way they choose. At Nature we publish only those papers that represent some significant conceptual novelty. Other journals might choose to publish anything that passes a simple technical review, and that’s fine.

    Do you need professional editors? Yes: scientists are used to studying very narrow fields in great depth. Editors are like journalists or reporters – they have a passing knowledge of a great diversity of things, knowing each one well enough to know whether it’s genuinely new or more of the same. Editors have the contextual knowledge that scientists lack – scientists have the technical knowledge that editors need to help them make decisions in particular cases.

    Why not have post-publication peer review?

    I’m not against post-publication peer review – I just think it’s never going to work as a wholsesale replacement for the system we have in place. Consider, as an analogy, the related example of print-on-demand publishing, in which authors can simply upload their work for sale onto Lulu or CreateSpace, without having the hassle of getting their work approved by agents, book editors and so on. Authors can do away with all that and reach readers directly through teh interwebs. There are loads of print-on-demand titles available, too many for anyone to be able to go through them all and sift them for quality. In the real world, readers don’t have the time or energy to go through all that, and if they can’t get a personal recommendation, they would prefer to rely on the imprimatur of a publisher, who will have done all the pre-selection.

    That doesn’t mean that every book that gets published is fantastic, because it’s not. Neither does it mean that there isn’t some literary masterpiece lurking in some PoD archive somewhere – because there is. It’s just nobody’s been able to find it amid the morass of not-quite-so-good PoD titles. As a result, PoD titles are generally ignored except in very, very specific cases. For example, I know an author who publishes works of interest to one Native American nation – a readership of a few hundred. In that case PoD is the only economic way to reach that audience. Therefore I think post-publication peer review might suit a few extremely specialized audiences, but will not succeed in the general market.

  11. Arsefuttock,

    Can’t find the one I’m looking for, but this will do nicely:- http://www.youtube.com/watch?v=htEyRM74VRQ