{"id":346,"date":"2011-01-18T09:55:14","date_gmt":"2011-01-18T07:55:14","guid":{"rendered":"http:\/\/occamstypewriter.org\/boboh\/2011\/01\/18\/more_ideas_about_ideas_in_ecology_and_evolution\/"},"modified":"2011-01-18T09:55:14","modified_gmt":"2011-01-18T07:55:14","slug":"more_ideas_about_ideas_in_ecology_and_evolution","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/boboh\/2011\/01\/18\/more_ideas_about_ideas_in_ecology_and_evolution\/","title":{"rendered":"More Ideas About Ideas in Ecology and Evolution"},"content":{"rendered":"

A couple of years ago I blogged about a new journal, Ideas in Ecology and Evolution<\/a>, and it’s experiments in the reviewing process. I was sceptical then, but happy to be shown wrong: I think we need these experiments to find out how . For this year, Ideas in Ecology and Evolution (IEE) is starting a new experiment, and this time they seem to have jumped the shark.<\/p>\n


\nThe experiment is announced in a new IEE editorial<\/a>. Briefly, they propose the following process:<\/p>\n

    \n
  1. The author solicits his\/her own referees, and can pay them for their trouble.<\/li>\n
  2. If revisions are suggested, the author does them and asks the reviewers to re-evaluate the manuscript.<\/li>\n
  3. Once the author has two favourable reviews, they sends the manuscript, with the reviews, to the journal, <\/li>\n
  4. The journal check with the reviewers that they have reviewed the submitted manuscript, and can solicit more reviews before reaching a decision on acceptance.<\/li>\n
  5. If the journal accepts the paper, the referees’ names are disclosed in the published version.<\/li>\n
  6. If the journal rejects the paper, it can be submitted to another journal, with the same reviews.<\/li>\n<\/ol>\n

    There are two innovations here: the author (not the editor) solicits reviews, and reviews can be re-used. The latter, recycling reviews, looks generally like a good idea – it’s not perfect but I’m sure we’d live with the problems. But it’s only going to work if journals will accept reviews from their competitors, and this is where the new IEE policy breaks down. Firstly because they’re the only journal doing this, and secondly because it’s unlikely any other journal would accept these reviews.
    \nThe problem is simply the idea that authors can invite who they want to review, and chose which reviews to send to the journal. And they can pay for the review! has whoever thought up that idea ever heard the phrase “conflict of interest”?
    \nThe first problem is that, despite the protestations of the IEE editors, it’s difficult to see how letting authors solicit their reviews will help. the editors think that it will lead to a greater incentive to reviewers:<\/p>\n

    Although the [IEE’s] model can work in theory without any requirement that referees be paid, paying referees – combined with published referee acknowledgement, plus opportunity for referees to post\/publish their commentary on the reviewed paper (in the event that it is published) – provides the important principal advantage of referee incentive.<\/p><\/blockquote>\n

    and thus to a greater overall quality:<\/p>\n

    Paid service combined with published referee acknowledgement, and opportunity for referees to post\/publish their commentaries on reviewed papers, would not only minimize referee bias and promote greater referee accountability, but would also engage referees more directly in the mission for discovery that the manuscript represents. Any worry that the [IEE’s] model might be inferior to the conventional process for manuscript merit judgment is unfounded when recognizing the currently limited record of success for reviewing panels of alleged experts (Wardle 2010).<\/p><\/blockquote>\n

    I’m not sure that money and having one’s name listed on the paper are a huge incentive. Several journals list the editor who took charge of a paper, but can you remember who the editor was of the last paper you looked at? If the paper is a real turkey, people might look at the referees’ identities, but then such a paper should be spotted by the referees and editors anyway.
    \nHaving your comments published seem like a incentive, but it obviously doesn’t count as a peer reviewed publication, so I wonder how much importance would be attached to it.
    \nBTW, the only evidence the editors give is the     Wardle paper, also published in IEE<\/a>. It’s not about pre-publication peer review, and only attempts to critique how the F1000 panel judged the importance of ecology papers. The relevance is dubious at best (and the paper itself isn’t terribly good either).
    \nWhat of the conflicts of interest? Well, the reviews are solicited by the author, and possibly paid for too by them too. The contract (whether formal or implicit) is between these two parties, so the reviewer’s responsibility is to the author, who wants a good review (i.e. one which will lead to acceptance of the paper. This may not mean uncritical, but that any criticism can be handled in revision), and not to the journal, who want an honest (and reasonably objective) opinion. It’s difficult to see how this improves matters. Authors will never ask Dr. McGit for a review, because they know they’ll get panned. and Dr. McGit can ask lowly students or post-docs to review, because (a) they need the money to feed their macaroon habit, and (b) they’re too scared of what he’ll say to his colleagues when they ask for a job.
    \nOK I’m exaggerating, but I think the point still holds: a referee will feel beholden to the author, and will want to please them, not a nebulous journal who may print their name in a year or two. Even if this bias isn’t conscious, I think it would still be there (anyone know of any studies, e.g. in psychology, that test this?).
    \nWhat makes it worse is that the author can select their reviewers, so if they don’t like a review, they can just ask for another one from someone else, and don’t tell the journal. I can’t see how a journal can protect against that.
    \nIf the journal is to retain any integrity, it has to label manuscripts for which authors solicited reviews, and if the reviewers were paid by the authors, this should be declared as a conflict of interest. And would you trust a review process which would allow this?<\/p>\n","protected":false},"excerpt":{"rendered":"

    A couple of years ago I blogged about a new journal, Ideas in Ecology and Evolution, and it’s experiments in the reviewing process. I was sceptical then, but happy to be shown wrong: I think we need these experiments to … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-346","post","type-post","status-publish","format-standard","hentry","category-science-publishing"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts\/346","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/comments?post=346"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts\/346\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/media?parent=346"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/categories?post=346"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/tags?post=346"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}