Last week a paper by Squazzoni et al appeared, which had analysed submissions to 145 scholarly journals to look for gender bias in acceptances and across the whole editorial process. They claimed not to find it. When I saw the headline I was puzzled. A careful analysis of their own publications by the Royal Society of Chemistry in 2019 had found significant evidence of bias across their editorial process, as I wrote about previously. Why the difference? Was this latest study different because it looked across multiple journals and disciplines, not just chemistry? Or was it different because the data they had access to provided less insight? My conclusion is the latter interpretation is correct. This recent study made some assumptions about where the problems might lie and focussed fairly narrowly there.
Squazzoni et al claim in their conclusion that
‘peer review and editorial processes do not penalise manuscripts by women’
and that, in some fields,
‘manuscripts written by women as solo authors or co-authored by women were treated even more favourably by referees and editors’.
This conclusion seems solely to be based on the fact that the acceptance rates were, in some cases, slightly higher for women than for men. Unfortunately, they excluded some key steps and issues in their study, presumably in part because they didn’t have access to them and in part because they didn’t fit the hypotheses they were aiming to test:
‘We concentrated on three possible sources of bias, i.e., the editorial selection of referees, referee recommendations, and editorial decisions, and examined all their possible relationships while controlling for important confounding factors such as journals’ field of research, impact factor, and single versus doubleblind peer review.’
So, they concentrated on some topics to the exclusion of others. The first step in the review process – did the editor even send the paper out for review, a huge gatekeeping event – was not explored at all. Yet, certainly for the highest impact factor journals such as Nature, this step is absolutely crucial. Many papers are simply rejected out of hand – but which ones and by whom? To conclude that editors treated women more favourably, at least in some cases, when this first hurdle is totally ignored seems somewhat naïve. Particularly so, when there is already evidence that this step has a detrimental effect. The RSC report states
‘we found that initial submissions from female corresponding and first authors are more likely to be rejected without peer review….’.
By excluding the challenge of this very first step, the recent study has to be regarded of dubious use as evidence that editors do not exhibit bias.
Another issue that neither they nor, indeed, the RSC considered was the length of the review process or how many revisions were requested before acceptance (or indeed rejection). Again, one can see how easily subtle bias can creep in here. For instance, if a male corresponding (or first) author is more likely to be given the benefit of the doubt on first submission when a similar woman is not but instead revisions demanded, then there will be a consequent delay for the woman. Tying in with this concern are the results of a study of NIH grant holders from a couple of years ago which looked at how men and women react to feedback about grant applications; it is hard to see why the conclusions should differ when it comes to referees’ comments. Their conclusion was
‘that gender bias in grant reviews (i.e., greater likelihood of highlighting inadequacy in reviews of women’s grants), along with gender differences in responsiveness to feedback, may contribute to women’s underrepresentation in academic medicine’.
It seems to me that referees’ comments about a manuscript are likely to have the same dampening effect on a woman’s motivation, confidence and hence career progression as comments about a grant application as this study demonstrated.
As yet another study showed, inappropriate referee comments are common, but are likely to have a much more damaging impact on minorities. These authors specifically identified that
‘future studies should test if receiving an unprofessional peer review leads to different acceptance outcomes depending on gender and/or race/ethnicity.’
but, as far as I know this has not been done anywhere. If an editor reads a negative review – however unwarranted – arising from gender bias, then they are more likely to require women to resubmit their manuscript than a comparable man. Receiving this more negative feedback because of bias in a referee, or being expected to go through multiple revisions because of how the editor reacts, will inevitably be disheartening, with the likely consequence that the discouragement will feed through into subsequent behaviour, even if ultimately the paper is accepted. Just looking at the crude statistics of acceptance is insufficient – as this most recent paper does – and may mask one of the subtle reasons why women are less likely to submit manuscripts, particularly to what are perceived as the ‘top’ journals.
Finally, it is inadequate to imagine that by seeking out more women referees the problem will be solved. As Melinda Duer and I wrote in the THE in early 2019,
“It is not just men who are biased against women; so too are women, as a 2012 study of job applications published in PNAS showed. So increasing the number of women in the reviewer pool, while giving more women useful experience, is unlikely to affect the number of female-authored papers accepted. Nor, correspondingly, is having more women on the editorial team of the journals.”
Nevertheless, this Squazzoni’s paper is enthusiastic about expanding the number of female referees and editorial board members (which may not be the ‘best’ use of any particular woman’s time), under the belief that then women will be more likely to submit manuscripts and participate more generally. To quote their abstract
‘However, increasing gender diversity in editorial teams and referee pools could help journals inform potential authors about their attention to these factors and so stimulate participation by women.’
This looks to me like a fig leaf, and smacks of fixing the women not the system. If women suspect there is bias in the process they may be right (as the RSC report shows).
For all these different reasons, I am not convinced by this study. It worries me that the paper will provide grist to those who want to believe the conclusions about a lack of bias in the editorial process, implicitly blaming women who are simply not able to ‘man up’ sufficiently, without thinking deeply about what the evidence actually demonstrates.
This resonates very strongly with me, especially your comments about how women may respond to feedback. When I first started submitting papers, I would always aim to address all the reviewers’ comments in full, including doing additional experimental work, no matter how tangential (or ridiculous– we’ve all been there). I think this was out of fear that the paper would be rejected if I didn’t do every last thing. However, a few years after starting my group, I started working on a project with a senior male colleague whose policy is to first try to rebut any requests for additional experiments with writing changes and or a well written response to reviewers. It had never occurred to me to even try this! And I’ve noticed he is almost always successful with this strategy. Editors and reviewers don’t really seem to insist or argue back in a second round of comments that the experiments must be done. I guess we can consider whether sometimes the reviewers have a point and the additional work would make the paper stronger, but it certainly leads to faster acceptance of papers.