{"id":2793,"date":"2015-04-29T09:06:30","date_gmt":"2015-04-29T08:06:30","guid":{"rendered":"http:\/\/occamstypewriter.org\/scurry\/?p=2793"},"modified":"2015-08-14T11:08:30","modified_gmt":"2015-08-14T10:08:30","slug":"amend-laws-scholarly-publication","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/scurry\/2015\/04\/29\/amend-laws-scholarly-publication\/","title":{"rendered":"Can we amend the laws of scholarly publication?"},"content":{"rendered":"

As part of its celebrations to mark the 350th anniversary of the publication of\u00a0Philosophical Transactions<\/em><\/a>, the world\u2019s longest-running scientific journal, the Royal Society has organised\u00a0a conference\u00a0to examine \u2018The Future of Scholarly Scientific Communication<\/a>\u2019. The first half<\/a>\u00a0of the meeting, held over two days last week, sought to identify the key issues in the current landscape of scholarly communication and then focused on ways to improve peer review. In the second half<\/a>, which will be held on 5-6th May, the participants will examine the problem of reproducibility before turning attention to the future of the journal article and the economics of publishing.<\/p>\n

The luminaries who assembled in the Royal Society\u2019s commodious headquarters in Carlton House Terrace included publishers, academics, representatives from funding agencies and learned societies, several journalists, and a smattering of FRS\u2019s and Nobel laureates \u2013 all well versed in the matter at hand. A report of the deliberations will be published in due course but I wanted to work through a central problem that surfaced repeatedly in the meeting last week.<\/p>\n

\"Tim<\/a><\/p>\n

The Royal Society\u2019s portrait of Tim-Berners-Lee, without whom we wouldn\u2019t be having this conversation.<\/span><\/em><\/p>\n

\u00a0<\/em><\/strong><\/p>\n

The central problem<\/em><\/strong><\/p>\n

The discussion on the first day \u2013 vividly live-blogged<\/a> by Mike Taylor \u2013 was an attempt to define the challenges facing scholarly publishing in the 21st century and covered territory that will be familiar to anyone who has read up on open access. The debate kept circling back to the same basic point: the over-weening influence of impact factors<\/a> and prestige journals, which have academics and publishers locked in an unhealthy relationship that seems increasingly unable to harness the opportunities presented by a digital world. It turns out that pretty much everyone can articulate the present difficulties \u2013 the hard bit is finding workable solutions.<\/p>\n

As I sat at the back of the room absorbing the proceedings, it struck me that the crux of the problem could be distilled into two laws of publishing. During the meeting I put them out on Twitter in a somewhat light-hearted way but there\u2019s a serious point here.<\/p>\n

First law:\u00a0<\/strong>To get maximum credit for your work, publish in a journal with a high impact factor.<\/p>\n

Second law:\u00a0<\/strong>For maximum speed of publication, choose a well-run mega-journal with \u2018objective\u2019 peer review*<\/span> or, better yet, a preprint repository.<\/p><\/blockquote>\n

The problem with these laws is that they are\u00a0incompatible and have given rise to unintended consequences, the central one being the creation of a system in which incentives for researchers are lashed to a slow and unwieldy process of publication. As presently configured academic publication is a competitive business in which journals vie with one another for the best quality science, or at least for the papers they think will boost their impact factors since that is one of the best ways to build a brand. As a result they have become the locus of the competition between researchers, who are now obsessed with the prestige points awarded by journals as the means to win jobs, promotion\u00a0or funding. The ensuing chase for publication in \u2018top journals\u2019 retards publication because researchers are willing to submit to multiple rounds of submission, rejection and re-submission as they work their way down the hierarchy of titles.<\/p>\n

There are other adverse effects (which will be considered below) but the solution to the problem has to be to weaken the coupling between the location of publication and the assessment of research and researchers.<\/p>\n

That is easier said than done. The coupling cannot be broken in a free market because of the competition between journals, which is of course fuelled by the competition between researchers. The competitiveness inherent in the system is not necessarily a bad thing since it can act as a spur for scientists to do their best work. The problem is that publication in a particular journal is too easily seen as the final<\/em> judgement on any given paper. If it\u2019s in\u00a0Nature <\/em>(to choose a prominent example), it must be world-class \u2013 such is the argument that I have heard repeatedly in common rooms, grant panels and hiring committees. This line of reasoning is seductive because there is some<\/em> truth in it \u2013 Nature <\/em>publishes a lot of great science. Unfortunately for too many people the argument contains enough truth for them to ignore the fine detail and from there it is a short step to take the journal impact factor as a proxy for the quality of a given piece of research. But it is vital to tease out and neutralise all the problematic aspects of our over-reliance on journal prestige. There are big wins to be had for scholarly publication and for researcher evaluation if we succeed.<\/p>\n

The simplistic but widespread view that good journals publish good papers is some distance from being the whole truth of the matter. Some papers get into Nature<\/em>\u00a0or Science\u00a0<\/em>because they catch a trendy wave but then fail the test of time \u2013 arsenic life<\/a>, anyone?\u00a0That\u2019s not to shout down journal predilections for trendy topics, which seem to be inevitable amid the ebb and flow of scientific tides but we should be mindful that they occur and calibrate our expectations accordingly. Some papers are published in top journals because they come from a lab at a prestigious university or one that has a good track record of publication and so may benefit from a type of hindsight bias<\/a> among editors and reviewers**<\/span>. Some get in because exaggerated claims or errors or fraud are not picked up in review. Again, that\u2019s not necessarily the fault of editors or reviewers \u2013 especially in cases of fraud \u2013 but we need to acknowledge that a system that determines rewards on the basis of publication in certain journals will\u00a0foster poor practice and fraudulent behaviour. This explains why retraction rates\u00a0correlate so strongly with impact factor<\/a>.<\/p>\n

Just as important, many papers are rejected that, with a different set of reviewers or a different editor might well have made the cut. Clearly the number of reviewers has to be restricted to keep the task of pre-publication review manageable***<\/span> but the stochastic nature of the decision-making process that results from relying on a small set of judgements is too often overlooked.<\/p>\n

The selectivity problem is exacerbated in top journals which restrict\u00a0entry for logistical reasons \u2013 the number of printed pages \u2013 and for brand protection. Nature<\/em>, for example, only has room for \u00a08% of the manuscripts it receives. Such practices presently act as an arbitrary and unfair filter on the careers of scientists because the reification of journal prestige or impact factor shapes presumptions and behaviour. Those who fail the cut on a particular day are likely to be disadvantaged in any downstream judgments. The mark of achievement is no longer necessarily \u2018a great piece of science\u2019 but \u2018a paper in Nature<\/em>, Science<\/em>, Cell<\/em> etc<\/em>\u2019. I\u2019ll say it again: these can often be\u00a0the same thing. But not in every case and best practice has got to be to always, always, always evaluate the paper and not the journal it is published in.<\/p>\n

No-one designed scholarly publication to have all these in-built flaws but the evident fact of the matter, widely agreed at the meeting, is that it has become a slow and expensive filtering service that distorts researcher evaluation and is no longer doing an optimal job of disseminating scientific results.<\/p>\n

As you see, I too can articulate the problem. But what about the solutions?<\/p>\n

\u00a0<\/em><\/strong><\/p>\n

Possible ways forward<\/em><\/strong><\/p>\n

For the first morning of the meeting I felt myself slipping into a funk of despair as the discussion orbited the black hole of the impact factor. It has become so familiar in the cosmology of scholarly publication that people are resigned to its incessant pull.<\/p>\n

However, that dark mood was slowly dispersed as occasional voices spoke insistently to press for answers. For me the bright spot of the conference was realising that the best way forward is likely to be in small pragmatic steps. Every so often one of these may turn out to be revolutionary. PLOS\u2019s invention of the mega-journal based on ‘objective peer review’ was one such case. So here are three more steps that I think are feasible, if all the key players in scholarly communication can be persuaded to really take on board the wrongness of journal impact factors as a measure of quality.<\/p>\n

ONE:<\/strong> Academics and institutions (universities, learned societies, funders and publishers) should sign up to the principles laid down in the San Francisco Declaration on Research Assessment<\/a>\u00a0(DORA). These seek to shift attention away from impact factors and to foster schemes of research assessment that are more broadly based and recognise the full spectrum of researcher achievements. Ideas for how to put such schemes in practice are available on the DORA web-site<\/a>.<\/p>\n

It is a real shame \u2013 and somewhat telling of the powerful grip of impact factors on scientific culture \u2013 that only a handful of UK universities have so far signed up to DORA (Sussex, UCL, Edinburgh and Manchester, at time of writing). I hope more will step up to their responsibilities after reading the warnings about the mis-use of the impact factor in the Leiden Manifesto on research metrics<\/a>\u00a0that was published in Nature<\/em> just last week. I would like to think it will become a landmark document.<\/p>\n

TWO: <\/strong>All journals with impact factors should publish the citation distributions that they are based on. The particular (and questionable) method by which the impact factor is calculated \u2013 it is an arithmetic mean of a highly skewed distribution, dominated in all journals by the small minority of papers that attract large numbers of citations \u2013 is not widely appreciated by academics. In the interests transparency, and to take some of the shine off the impact factor, it makes sense for journals to show the data on which they are based, and to allow them to be discussed and re-analysed. Nature Materials<\/em> has shown that this can be done<\/a>. I invite journals that might object to a move that would improve authors\u2019 understanding and help to promote fairer methods of researcher assessment to explain their reasoning.<\/p>\n

The difficulty with this proposal, as pointed out at the meeting, is that the data are amassed by Thomson-Reuters<\/a> who have so far refused to release them. This practice can now be declared contrary to the 4th principle of the Leiden manifesto<\/a>, which holds that data collection and analytical processes should be open, transparent and simple. If Thomson-Reuters are not interested in being part of the solution to the problem of impact factors, they have to be side-lined. Perhaps there are ways to crowd-source the task of assembling journal citation distributions? I would welcome suggestions from tech-savvy readers as to how this might be achieved in practice.<\/p>\n

THREE:\u00a0<\/strong>The use of open access pre-print servers should be encouraged. These act as repositories for manuscripts that have yet to be submitted to a journal for peer review and have the potential to short-circuit the delays in publication caused by the chase for journal impact factors.<\/p>\n

I\u2019m not familiar with the situation in chemistry but know that preprint servers have come rather late to the life sciences, inspired by the\u00a0arXiv<\/a> that has been operational in many sub-disciplines in physics and maths since 1991. bioRxiv<\/a> (pronounced \u2018bio-archive\u2019) has 1300 preprints, while the PeerJ Preprints<\/a> tally has just passed 1000. These are low numbers \u2013 the arXiv has over a million depositions \u2013 but they are growing and there was such widespread support for preprints at the Royal Society meeting that I began\u00a0to get a sense that they could be transformative.<\/p>\n

Preprints speed up the dissemination of research results (albeit in an early form) \u2013 and that is likely to be healthy for the progress of science. Uptake will increase as more researchers discover\u00a0that fewer and fewer journals<\/a> disallow submissions that have been posted as pre-prints.\u00a0Nay-sayers might contend that there are risks in disseminating results that have not been tested or refined in peer-review but such risks can be mitigated by open commentary. As noted in the meeting by\u00a0bioRxiv\u2019s Richard Sever, preprints tend to attract more comments than published papers because commenters feel that they can make a positive contribution to shaping a paper before it is published in its final form.\u00a0Such open review practices have been harnessed successfully<\/a> by the innovative open-access journal Atmospheric Chemistry and Physics<\/em><\/a>.<\/p>\n

Preprints may not solve the problem of impact factors, since the expectation is that most or all will eventually submitted for competitive publication in a journal; however, they could offer some mitigation of their worst effects, especially if authors were to be recognised and rewarded for the speed at which they make their results available. That\u2019s something for funders and institutions to think about. Moreover, because they have digital object identifiers and are citable objects, they provide a rapid route for establishing the priority of a discovery and a cost-effective way to publish negative results<\/a>.<\/p>\n

 <\/p>\n

Finally\u2026<\/em><\/strong><\/p>\n

There is more to say and more dimensions of the arguments made here to tease out but I will leave things there for now. All of the above may be no more than the optimistic after-glow of the meeting but I submit my three proposals for serious consideration. I hasten to add that\u00a0none of them are\u00a0original, and I know that I\u2019ve written before about pretty much everything discussed above \u2013 but we have to keep these issues alive. To paraphrase Beckett, it\u2019s our only hope of finding ways to fail better.<\/p>\n

\u00a0<\/span><\/p>\n

Notes<\/em><\/strong><\/span><\/p>\n

*\u2019Objective peer review\u2019 was the term adopted at the meeting to describe the type of peer review pioneered by PLOS ONE<\/a><\/em>, which will accept papers that report an original piece of science that has been performed competently. The impact or future significance of the work are not considered. I am not comfortable with the term since this type of review is clearly not \u2018objective\u2019 but it will have to do for now.\u00a0<\/span><\/p>\n

**Nature<\/em> has recently introduced a double-blind review<\/a> to combat this. However, it is optional for authors and I find it hard to imagine a group that has a history of publication in Nature<\/em> opting for it.<\/p>\n

***Richard Smith (formerly editor of the BMJ<\/em>), who spoke powerfully at the meeting to criticise peer review<\/a>, and proponents of post-publication peer review will\u00a0no doubt take issue here.\u00a0<\/span><\/p>\n

\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

As part of its celebrations to mark the 350th anniversary of the publication of\u00a0Philosophical Transactions, the world\u2019s longest-running scientific journal, the Royal Society has organised\u00a0a conference\u00a0to examine \u2018The Future of Scholarly Scientific Communication\u2019. The first half\u00a0of the meeting, held over … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[152,107],"tags":[],"class_list":["post-2793","post","type-post","status-publish","format-standard","hentry","category-open-access-2","category-scientific-life"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2793","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/comments?post=2793"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2793\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/media?parent=2793"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/categories?post=2793"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/tags?post=2793"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}