{"id":237,"date":"2011-01-26T21:57:24","date_gmt":"2011-01-26T21:57:24","guid":{"rendered":"http:\/\/occamstypewriter.org\/notranting\/?p=237"},"modified":"2011-01-26T21:57:24","modified_gmt":"2011-01-26T21:57:24","slug":"more-peer-review-zzzzzz","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/notranting\/2011\/01\/26\/more-peer-review-zzzzzz\/","title":{"rendered":"More peer review. Zzzzzz."},"content":{"rendered":"

In which, late as ever, I remember to say something about peer review.<\/span><\/p>\n

There has been a bit of a flurry of commenting about peer review around OT just recently, with both RPG<\/a> and Girl, Interrupting<\/a> weighing in. The general consensus with them, and other bloggers, here, and most commenters on their posts, would be something like:<\/p>\n

\u201cPeer review: imperfect but necessary \u2013 and better than the alternatives\u201d.<\/strong><\/span><\/p>\n

Now, I guess that would also broadly sum up my own view. I would add, I think, that peer review is more of a problem for grants than for papers. But, of course, this is more a function of the ludicrous success (or rather failure) rates for grant applications than of the refereeing process. More on that sort of stuff from me here<\/a>, and in Stephen Curry’s recent post here<\/a>.<\/p>\n

Anyway, as I am feeling guilty (as usual) about not posting anything here for a while, I have disinterred what I wrote about peer review for an editorial in Physiology News<\/em> a couple of years back (Autumn 2008, to be precise). Somewhat pompous and self-consciously \u201cbalanced\u201d<\/span><\/em> style, yada yada, but you\u2019re used to that by now.<\/p>\n

—————————————————————————————————————–<\/p>\n

Peer review<\/strong><\/p>\n

Peer review, which lies at the heart of the scientific process, has a long history; the style of (‘single blind’) anonymous peer review now in common use was described by the Royal Society of Edinburgh in 1731, though the basic idea is far older. Despite this, widespread application of a system of anonymous expert referees did not become commonplace until later than is often believed, around the middle of the 20th century. Expert peer review in some form is now pretty much universal in scientific journals. So is there anything new to say?<\/p>\n

Science vs non-science<\/strong><\/em>
\nOne point that scientists sometimes need to remember is that peer review is not just important ‘internally’, but externally too. Properly functioning peer review is a key way to distinguish science from non-science (nonsense?). In a world where we are bombarded by apparently scientific claims \u2013 often for things that are being sold to us \u2013 it is important to have ways of telling sales talk and science apart, and peer review is one. As Sense About Science put it, peer review is an \u2018essential arbiter of scientific quality\u2019<\/span>(1).<\/p>\n

A fly in this ointment, of course, is that there is peer review and peer review. Journals in the top couple of dozen, or possibly more, journals in established subject categories \u2013 such as ‘physiology’ \u2013 maintain rigorous review processes, as we all regularly experience. But there are a lot of journals, and reviewing standards vary widely. As a recent report for the Publishing Research Consortium puts it (2):<\/p>\n

\u2018Because the peer review standards of different journals vary, it is widely believed [by scientists] that almost any genuine academic manuscript, however weak, can find a peer-reviewed journal to publish it if the author is persistent enough.\u2019<\/span><\/p>\n

So, while publication in a peer review journal is some kind of quality mark, there is a blur at the edges.<\/p>\n

Bibliometrics \u2026 inevitably<\/strong><\/em>
\nOne way we might seek to clarify the reliability of work published in peer review journals are journal \u2018pecking orders\u2019, nowadays often substituted by bibliometric rankings. The most common of these, journal impact factors (IFs), while derided regularly and with plenty of justification, are clearly here to stay. As The Physiological Society\u2019s journal editors put it a couple of years ago:<\/p>\n

\u2018There is considerable divergence of opinion about the significance of [impact factors as a] measure of journal quality, but author surveys continue to confirm that a journal\u2019s IF is among the most important considerations in choosing where to publish.\u2019<\/span><\/p>\n

That is, impact factor is a proxy for pecking order, and thus perhaps, to a limited extent, for reviewing standards. This may indeed be the only thing for which journal IFs are useful, since their worthlessness in assessing individual scientists and their work has been attested repeatedly, most \u00a0trenchantly by David Colquhoun (3).<\/p>\n

Of course, journal IFs are only useful even in this restricted fashion within subject categories; their inadequacies when NOT comparing like with like are well described. While general science journals like Nature<\/em> and Science<\/em> have far higher IFs than journals like J Physiol<\/em> and J Gen Physiol<\/em>, I have yet to meet a single scientist in physiology who thinks that the reviewing at the general journals is more rigorous. To take another example, comparative physiology journals have notoriously low IFs, though the technical quality of the work they publish is high.<\/p>\n

Can we improve anything about peer review?<\/strong><\/em>
\nWhile peer review is clearly vital, it is also imperfect; I suspect there is no scientist alive who does not have some complaint to recount. To their credit, journals try hard to \u2018audit\u2019 their review processes, and to tweak them in ways that will help them work better. The latest fashion seems to be for more explicit instructions to reviewers as to how to assess a paper and write a reviewer\u2019s report. David Linden, the editor of the Journal of Neurophysiology<\/em>, writes about this here (4), and\u00a0Nature Cell Biology<\/em> has also weighed in (5).<\/p>\n

One thing I was glad to see in the latter was the exhortation to reviewers to ask \u2018Are all claims made supported by the data?\u2019 <\/span>A personal view is that reviewers tend to be harder on whether they think experiments are technically correctly done \u2013 as Nature Cell Biol<\/em><\/span> puts it: \u2018Are key\u00a0experiments or crucial controls missing? Are the data significant and definitive?\u2019 <\/span>\u2013 than on the inferences authors subsequently draw from the data. Though this is understandable, the\u00a0danger here is that if an author repeats the same rather tenuous extrapolation in several published papers, they can then conceivably write a review citing all these papers and boosting tenuous extrapolation to the status of \u2018well-tested hypothesis\u2019<\/span> \u2013 next stop the textbooks. So a personal plea would be for reviewers to pay a bit more attention to what authors say in the Discussion, as well as in the Methods and Results.<\/p>\n

This also brings me to a final point \u2013 what happens if you see a paper that clearly has something in it that is wrong, but that the journal\u2019s referees have missed? It is certainly possible to write to the journal or its editors, though such letters rarely, in my experience, see the light of day.<\/p>\n

My preferred solution to this problem is an electronic response thread following the online version of the article, something favoured by some medical journals and increasingly by exclusively
\nonline journals. While it is perfectly possible to pen a short review commenting on a paper or papers, this rarely happens either, mostly for \u2018activation barrier\u2019<\/span> reasons. Writing a review, even a short one, is hard work. In contrast, penning an e-letter is a lot easier, and in the best cases generates quite an informative online debate. It is a bit like seeing the reviewing happening\u00a0live, except after publication, and sometimes shows up things the journal reviewers missed. Another recent Nature Cell Biol<\/em> editorial reveals that response threads will be coming soon for the\u00a0Nature Group journals (6) \u00a0[Note: now in place, since this was written 2 yrs ago], and it will be interesting to see who else follows suit.<\/p>\n

So to peer reviewers out there: keep up the reviewing standards, watch the authors\u2019 extrapolations \u2013 and see you in an electronic response thread soon. Even after a few hundred years, this is no time to get sloppy.<\/p>\n

1 “Peer review\u201d<\/a>. Sense About Science.
\n2 Peer Review: Benefits, perceptions and alternatives.<\/a> Publishing Research Consortium.
\n3 Colquhoun D (2007). How to get good science. Phys News<\/em> 69, 12\u201314. [online on his blog here<\/a>]
\n4 Linden DJ (2008). Warm, fuzzy feeling. J Neurophysiol <\/em>100,1.
\n5 Good review (2008) Nature Cell Biol<\/em> 10, 371.
\n6 What to publish? (2008). Nature Cell Biol<\/em> 10, 247<\/p>\n

[More hotlinks later if I can be bothered!]<\/p>\n

————————————————————————————<\/p>\n

So what would I add to that today? <\/em><\/strong><\/p>\n

Well, one thing is that, like the new President of the Royal Society in his Horizon<\/em> TV programme the other night<\/a>, I stand by the second paragraph \u2013 we need<\/em> peer review. To those like me who spend a fair amount of time combating pseudoscience, it is absolutely clear that \u201cserious\u201d peer review, along the lines described above, remains one of the major differences between real science and the cargo cult stuff<\/a>.<\/p>\n

Unfortunately, the practitioners of many pseudosciences have got wise to this, and have set up their own journals. These too practice a kind of peer review \u2013 it is just that there it is review by your real<\/em> peers<\/a>, namely other people who have suspended their skepticism and critical faculties. Often this is done unknowingly, with people fooling themselves in their desire to believe – a kind of confirmation bias<\/a>. But it is nonetheless a major problem, certainly in the area of complementary and alternative medicine (CAM).<\/p>\n

There are certainly people who feel traditional pre-publication peer review should be replaced by something else, like \u201ccrowd source\u201d<\/span> peer review, or post-publication review. My friend David Colquhoun occasionally muses along these lines, and there are also vocal proponents like Cameron Neylon<\/a>. I don\u2019t think I am convinced. I am a fan of the idea of post-publication \u201ccrowd\u201d review \u2013see my words above. But I have my doubts about whether it is really workable.<\/p>\n

One reason is that journals that do run online comments threads to allow \u201ccrowd response\u201d<\/span> tend to have rather little in the way of comments on many papers. Nature<\/em>, for instance, only seems to attract comments on articles when the topic is something that gets the trolls and nuts in a frenzy, at which point the ratio of insane to rational comments tends to be about 10 (or more) to 1. For examples try, for instance, anything mentioning the fringe types who do not HIV is the cause of AIDS. That is not crowd review \u2013 more like spittle-flecked shouting.<\/p>\n

The only journal I have seen thus far with an intermittently successful online comments set-up is the British Medical Journal<\/em>, and even there it is patchy. Some articles do attract reasoned critique. Others, again, attract only the green ink brigade. Or, in the case of anything relating to vaccines, end up back where we just noted with Nature<\/em> threads and HIV.<\/p>\n

So in the end, I am back with the analogy that makes Cameron Neylon so tetchy<\/a>. Yes, that Churchill one, about peer review being imperfect, but less imperfect than the alternatives.<\/p>\n

Though we were discussing this after RPG\u2019s post on peer review<\/a> the other day, I hadn\u2019t tracked down the real source of Churchill\u2019s actual remark, which I will quote.<\/p>\n

\u201cIndeed, it has been said that democracy is the worst form of government except all those other forms that have been tried from time to time\u201d<\/span><\/p>\n

Per Wikiquotes<\/a>, and via Cameron Neylon’s blog, the source is The Official Report, House of Commons \u2013 otherwise known as Hansard – (5th Series), 11 November 1947, vol. 444, cc. 206\u201307<\/p>\n

Cameron Neylon makes the interesting point<\/a> that the analogy we make between this and peer review is flawed, since alternatives to traditional peer review have not really been tried in the modern scientific era. And I think I would accept that that is a good reason to trial<\/em> some alternatives, in a limited way. But until there is a really compelling body of evidence suggesting we dump pre-publication peer review, I shall be sticking to my take of Faute de mieux<\/a><\/em>.<\/p>\n","protected":false},"excerpt":{"rendered":"

In which, late as ever, I remember to say something about peer review. There has been a bit of a flurry of commenting about peer review around OT just recently, with both RPG and Girl, Interrupting weighing in. The general … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":13,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,25,5,7,9],"tags":[],"class_list":["post-237","post","type-post","status-publish","format-standard","hentry","category-history","category-peer-review","category-pseudoscience","category-science-policy","category-the-life-scientific"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/posts\/237","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/comments?post=237"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/posts\/237\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/media?parent=237"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/categories?post=237"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/notranting\/wp-json\/wp\/v2\/tags?post=237"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}