I’ve been asked to write something for the BES newsletter on double-blinding for journals. So I thought I would try out this Web2.0 thingy, so here’s a draft for you to pick over and comment on. It’s a bit too long, so most suggestions for where to cut will be welcome. Oh, and I couldn’t work out how to have multiple footnotes on the same word, but I hope you can work it out.
The traditional single-blinded approach to journal review is in a perpetual state of criticism. One perceived problem is that referees make their assessments based on the identity of the author(s), rather than just on the quality of the paper. This could bias the reviews if the decisions are based on, for example, the gender of the author or on whether the author is well known. An obvious fix for this problem is to double-blind a review: i.e. the reviewers are simply not told the identity of the authors. Sounds great, and a recent survey found that scientists liked the idea, but on further inspection things get more complicated.
We all hear anecdotes about referees holding up papers, or giving bad reviews of papers that disagree with them, but is there more than just anecdote? Firstly, there is at best little evidence for a difference in the quality of1 reviews2. Secondly, one argument that has been raised for double-blinding is that it removes biases against women authors. But again, there is no evidence for an3 effect4 (this last conclusion has been contended5, but oddly without contending the analyses and data on which it is based).
So there is little evidence that double-blinding is better for journals. Even worse, the evidence is pretty clear that it does not even work: the proportion of referees that correctly identify the author has been estimated at 322 and 463 percent. Of course, an author is more likely to be recognised if they are known to the reviewer, but this is the same factor that can bias a reviewer in favour of the author. Thus, the two biases may even act together to make the situation more complicated6.
Should journals use double-blind reviewing? There is little evidence that it is beneficial, and the overheads, in terms of ensuring effective blinding, are significant. From the point of view of a journal, the argument in favour is that it might attract better manuscripts: it does not matter if double-blinding has an effect on quality, it only matters that it is perceived to have an effect, and that some authors will chose to submit to a journal because they think they will be more successful. Cynical, I know.
1 van Rooyen, S., Godlee, F., Evans, S., Smith, R., Black, N. (1999). Effect of Blinding and Unmasking on the Quality of Peer Review. J. Gen. Intern. Med. 14:622–624.
2 Justice, A.C., . Cho, M.K., Winker, M.A. Berlin, J.A., Rennie, D. and the PEER Investigators (1999) Does Masking Author Identity Improve Peer Review Quality?: A Randomized Controlled Trial. JAMA. 280: 240-242.
3 Blank, R.M. (1991) The effects of double-blind versus single-blind reviewing: experimental evidence from The American Economic Review. The American Economic Review 81: 1041-1067.
4 Webb, T.J. et al. (2008) Does double-blind review benefit female authors? TREE, in press. DOI: 10.1016/j.tree.2008.03.003
5 Budden, A.E., Lortie, C.J., Tregenza, T., Aarssen, L., Koricheva, J., Leimu, R. (2008) Response to Webb et al.: Double-blind review: accept with minor revisions. TREE, in press. DOI: 10.1016/j.tree.2008.04.001
6 As I found out. Blank (1991) has evidence that middle-ranked institutions fare worse under double-blind review, which could be used to support my conclusions, but it looks too neat – I would want to see more studies before declaring success.
I’m shocked that it’s as low as 42% identification. The other 58% must have been too lazy to read the Introduction, where the authors almost always spend a paragraph describing how the current work follows on from previous work the lab has published.
This system would only work if the authors had to remove all references to their own work when they introduce why the study was initiated. And I fear the paper would suffer as a result; it’s hard enough sometimes to work out why authors have taken the course they have even when they do introduce the narrative.
That is interesting, Jennifer. For double-blinded review to deserve that name, one would not be allowed to self-cite. This getting quite complicated, as we now may be merging into the topic of citation index being boosted by self-citations. Martin, we need your thoughts here!
I don’t think ISI count self-citation in the impact factor, Massimo.
and what about calculation of h-index?
It’s not just self-citation Massimo, it’s the actual work that leads up to that point.
“Someone showed that calcium is released from intracellular stores when they are tickled with girrafes, but we’re not (allowed to) telling who”.
Doesn’t quite work for me.
Yup, I can see your point, Richard (and that of Jennifer). In order to comply with double-blinded review, one should enter a state of denial of the foundations of the work that is being submitted. That’s bad.
With some work, I think the authors have already entered that state, Massimo…
It’s pretty short as it is, no?
I’d shuffle things around a little, but it loses some informal tone. Keeping the same 1st paragraph:
Is there more than anecdotal evidence that referees hold up papers, or give bad reviews to known authors who disagree with them? So far, there is little data supporting a qualitative difference in reviews when author identity is masked [1,2].
A specific argument in favour of double-blinding is that it removes biases against women authors. But evidence for this effect [3,4] is at best contentious [5], and until the data on which it is based is fully re-analyzed, the jury is still out.
From the point of view of a journal, double-blinding might attract better manuscripts: whether or not there is a real effect on the subjective nature of reviews, a perceived effect may lead some authors to submit to a journal because they think they will be more successful.
However, whether or not it is beneficial to journals, double-blinding does not even work: the proportion of referees that correctly identify the author has been estimated at 32 [2] and 46 [3] percent. An author is more likely to be recognised if they are known to the reviewer, but this is the same factor that can bias a reviewer in favour of the author. Thus, the two biases may even act together to make the situation more complicated [6], in particular for niche specialties with small followings.
So, should journals propose double-blind reviewing? There is little evidence that it is beneficial, and the overheads, in terms of ensuring effective blinding, are costly and difficult to define.
Hey, you asked for it. It’s not so much shorter (very slightly) as ordered a bit differently.
I agree with the points you make in your letter. You could make this passage a bit shorter:
Secondly, one argument that has been raised for double-blinding is that it removes biases against women authors. But again, there is no evidence for an3 effect4 (this last conclusion has been contended5, but oddly without contending the analyses and data on which it is based).
Suggested change:
Second, it has been suggested that double-blinding removes biases against women authors, but there is no clear evidence for an effect (3-5).
I think “cliquiness” and geographical biases are also worth mentinioning. In the case of the former, journals with editorial boards and/or editors who are themselves practicing scientists are, on the face of it, susceptible to this bias. Then, you would be looking at triple-blind (editor/author/referees) not just double, even more complex, expensive and error-prone for a journal to control for.
“Geographical” bias can be both against authors and against referees.
You can’t get into all this in a short letter, so how about:
Your version (paragraph 1):
This could bias the reviews if the decisions are based on, for example, the gender of the author or on whether the author is well known.
To something like:
This could bias the reviews if the decisions are based on, for example, the gender of the author; their geographical origin and/or place of work; or on whether the author is well known and/or does work supporting the views of the academic editor or editorial board.
I think that this addition, though it makes the letter longer, conveys a range of problems with identifying and eliminating bias in peer review, rather than the present emphasis of the letter on one factor (gender).
Thanks for your suggestions, Heather and Maxine. The revised versions look much better. Now I have to combine them…
I do have one complaint – the replies interrupted me when I was
following the cricketrevising a manuscript.Cliquiness is something I hadn’t thought about, but I agree it’s worth mentioning. I also wondered about mentioning the role of editors, but decided there wasn’t space. It’s also a topic I haven’t explored enough yet to have any firm opinions (other than that Nature editors are the pinnacle of their profession, of course).
On editors, I think there is a definite difference between the role of a “professional” editor (Nature, Science, Cell etc) and an “academic” editor (a researcher doing a stint at a society journal for the field).
And it is the editors in each case who choose the peer-reviewers!