A lot has been said about peer review, recently by Frank here, here and here, by Richard here, here and here, by Sylvia and by others. So what more can I add?
Like democracy, it’s not ideal, but compared the the alternatives–well it’s the best we can do. So I’m not out to undermine peer review. I’m out to undermine journals that masquerade as being peer reviewed.
A lot has been said about some of the new journals that keep cropping up, in particular the lack of scrupulous principles with regards to publishing. But I would like to point out a “loophole” of sorts, in a widely recognized journal.
The Proceedings of the National Academy of Sciences, USA, has been widely considered a prestigious journal in which to publish papers. Some of this aura, methinks, comes from the old days when publishing was not as difficult, and peer review was not as rigorous. National Academy of Science members were allowed to directly “contribute” manuscripts that were automatically accepted and published.
Sound anachronistic? It is. Sound like an evolutionary throwback to the Soviet 70s? It does. But hey–guess what? This system is still in place today. If you didn’t know that, you do now. And guess what again? It’s actually gotten WORSE. Indeed.
Now not only are “Contributors” (National Academy of Science members) allowed to submit 4 papers a year to be automatically accepted and published, but there is another even more disturbing track. I’ll come to that in a minute. But first, the direct contributions, as they are called.
From the PNAS submissions instructions:
An Academy member may submit up to four of his or her own manuscripts for publication per year. To contribute an article, the member must affirm that he or she had a direct role in the design and execution of all or a significant fraction of the work and the subject matter must be within the member’s own area of expertise. Contributed articles must report the results of original research. [SKIPPING A FEW LINES HERE ABOUT FINANCIAL DISCLOSURES, AND THEN…] When submitting using the contributed process, members must secure the comments of at least two qualified reviewers. Reviewers should be asked to evaluate revised manuscripts to ensure that their concerns have been adequately addressed. Members’ submissions must be accompanied by the names and contact information, including e-mails, of knowledgeable experts who reviewed the paper, along with all of the reviews received and the authors’ response for each round of review, and a brief statement endorsing publication in PNAS.
Did everyone catch that? The authors are responsible for obtaining their own reviews. They decide who the reviewers should be, contact them directly and obtain the critiques. Are you, as a PI, going to reject a National Academy of Science member’s manuscript from PNAS if you are asked to review it? It’s a great way to make friends! So is this really peer review? When your peer is aware that you, as a National Academy of Science (NAS) member, will be quite cross with her/him if you dare to make serious critiques (not to mention reject the manuscript)?
Okay–I know that it’s certainly not trivial to become a NAS member. Most of these researchers have certainly been chosen due to their long careers of excellent science. Many of them choose NOT to publish in PNAS because they know it is not viewed highly in some circles. But in can be used as a “dumping ground” for papers that have been unable to get into real peer reviewed journals.
Consider this, though. There is another track–a relatively new track–that PNAS allows, that in my view is even worse than the NAS contributor mode: It’s called “Direct Submission.” What does this mean? It means that the authors have secured in advance a”pre-arranged editor”? Oh–that smacks of a Soviet era style “ole boys network.” Find an editor in advance–a friend, colleague, mentor, brother, sister–someone who will agree in advance to get the paper published. Have a look at this, again from the PNAS submission site:
Prior to submission to PNAS, an author may ask an NAS member to oversee the review process of a Direct Submission. Prearranged editors should only be used when an article falls into an area without broad representation in the Academy, or for research that may be considered counter to a prevailing view or too far ahead of its time to receive a fair hearing, and in which the member is expert. If the NAS member agrees, the author should coordinate submission to ensure that the member is available, and should alert the member that he or she will be contacted by the PNAS Office within 48 hours of submission to confirm his or her willingness to serve as a prearranged editor and to comment on the importance of the work.
Now this actually manages to get around not one, but two levels of review. After all, for the ordinary-person’s peer review track, the editorial board/editor generally rejects 75% of the incoming papers without their even reaching peer review. The “pre-arranged editor” trick circumnavigates the need to go through this initial triage selection process, and shunts the paper directly into press.
Pretty amazing, eh? All you have to say is that there isn’t enough general expertise on the board, or that the paper is–how do they put it? Here it is: Counter to a prevailing view or too far ahead of its time to receive a fair hearing. So if your paper is contrary to current views or “ahead of its time” (what the hell is that supposed to mean–and who decides this anyway?)–get a free pass. But the catch? You need to have a buddy on the editorial board. Otherwise, who will do this for you. You need to be part of the “ole boys network.”
Doesn’t everyone have a disclaimer these days? After all, you don’t want to be sued. There is a statement in the submission site that says the following:
“Papers with a prearranged editor are published with a footnote to that effect.”
Well, why not be more explicit? These papers are not peer reviewed and should be treated as such.
As for this journal: it’s time to move into the 21st century.
Steve – the idiosyncratic reviewing process at PNAS certainly leaves it open to the problems you describe, but this of course does not mean that all PNAS papers are poorly reviewed or flawed. (Declaration here, having published half a dozen papers in PNAS). My own experience, and I am not a member of the NAS, is that whether or not I’ve gone via an academy member, the reviewing has been extremely stringent on every occasion. I’ve also had one or two submissions that scandalously didn’t even get through triage.
@stephenemoss
I agree. It certainly doesn’t mean that all PNAS papers have not undergone proper review or are flawed–but it leaves it up to the discretion of the particular editor that you contacted. What this means is that your contact(s) is a principled person (perhaps extensively so, if your papers are actually getting rejected!).
At the same time, you will admit that this is not exactly a policy that tends to promote fair review for all. Scientists who don’t have personal contacts on the editorial board are already at a distinct disadvantage in even getting their papers sent out for review.
My first paper as a graduate student was with PNAS. My mentor sent it to her post-doctoral mentor, a NAS member, fearing that he would be even harsher with her work. He wrote to her that he was sending it out to 2 reviewers and would accept their verdict. It was eventually accepted with a lot of work and two revisions. Nonetheless, I can’t agree that this is a fair way for manuscripts to be judged.
Correction: my first paper as a Ph.D. graduate student, as I had published a paper from my Msc research…
Steve – if there’s one thing I cannot abide it is editorial laziness, so when I recently had a paper fail the PNAS triage I wanted to know why. The reason I was given, ‘…paper more suited to a specialised journal’ just didn’t wash because PNAS had published a string of papers on precisely the same subject in the last few years. I challenged their decision, and asked for an explanation. None was forthcoming. The paper is now in the hands of a journal that at least takes its editorial responsibilities seriously.
Steve- A few years back, I tried a submission to PNAS via the regular “drone” track for all those who are not part of any “old boys network.” Despite taking very careful pains to point out why precisely this manuscript was of high general interest (coupled with the fact that I tend to be very conservative about the journals to which I submit), the manuscript was tossed back to in record time with the explicit comment “not enough interest for the general reading public.”
Well I know that every author thinks his/her paper is always novel and exceptional, but as I noted, I have a pretty fair grasp of what really is beyond just adding another brick to the wall (which of course, can be excellent and first-rate science). And my “proof?” Sent the manuscript out 24 h later to Journal of Biological Chemistry as a “Report” (defined as the top 5% of all JBC papers and having significant and immediate interest), where it was rather quickly accepted with notable comments by the reviewers about the general interest.
But these are general issues that occur in many journals that reject based on the abstract without actual review. PNAS has the added layer of their weird direct submission and pre-arranged editors, which in my view is simply anachronistic.
Thanks for highlighting the potential problems with this journal’s submission and review process. I’m sure stephenemoss is right to point out that there are many very good papers published in PNAS that have been through a stringent and entirely above board process of peer review.
Just to put some meat on the bones of the concerns Steve Caplan raises, however, I wrote a couple of posts earlier in the year about a paper with serious flaws that went through the “pre-arranged editor” route to PNAS publication:
http://j0ns1m0ns.blogspot.com/2011/02/exercise-may-be-good-for-you-but-it.html
A couple of months later, PNAS published some scathing correspondence from other researchers pointing out the major errors in the original paper:
http://j0ns1m0ns.blogspot.com/2011/04/update-on-exercise-and-memory-story.html
This is only one example of an obviously flawed paper that has managed to get through the publication route Steve highlights and, as I say, I’m sure there are many that are entirely above reproach. But it is enough for me to be very concerned about the veracity of anything I read in that journal.
Jon,
Thank you for linking to your blog and for the excellent example. I did remember hearing about it, but my memory these days–ah well, I guess I need more exercise…
Seriously, though, although PNAS explicitly notes that “Prearranged editors should only be used when an article falls into an area without broad representation in the Academy, or for research that may be considered counter to a prevailing view or too far ahead of its time to receive a fair hearing, and in which the member is expert.”
I don’t want to start with any specific examples, but a very cursory look through the accepted papers with a “pre-arranged editor” footnote shows that in many cases the editor is not an expert in the field.
Hi Steve,
I’ve known about the PNAS alternative submission routes for years (and therefore check how individual articles have come through when evaluating them), but hadn’t seen the ‘Direct Submission’ route before. I guess that the handling editors are strongly encouraged to remain principled when dealing with such submissions, as it would be too easy to ridicule the journal (and Academy) for having such an easily abused system.
PNAS is highly variable across subject areas. In my own area (Ecology/Population Biology), it doesn’t have a very good subject specific impact factor*, IIRC. I’ve come across a couple of recent examples of pretty poor papers published through the ‘traditional’ route, so this isn’t a guarantee of quality either.
In general, I’d say it’s worth remembering that it’s science we’re dealing with when evaluating any contribution, not {insert your favourite} god’s holy law. Richard Feynman (a mere demi-god) found some aspects of the Academy sufficiently undesirable to lead him to resign, after all. I don’t imagine the Direct Submission route would have encouraged him to sign up again though…
* yeah – we all know IF isn’t a good measure of individual paper quality, but it’s a start.
Everything I’ve read written by Feynman suggests an integrity that would not be reconciled to the type of submissions that PNAS allows.
Is there a separate IF for “pre-arranged editor” papers as opposed to real submissions? There should be. In fact, they should even change the font, format and style of these papers. If they really wanted to support the journal, they would simply call them the non-reviewed track .
I have enormous admiration for Feynman for this and many other reasons. I wish I could be as principled as he was, but given the pressures on getting and maintaining funding these days, sometimes a backdoor approach is too appealing to turn down. Basically, the system we work in means that few researchers now can afford to be so principled. This ultimately makes science a little bit harder than it needs to be for everyone
I agree that in today’s climate, any little edge is something to grab onto. All the more reason for the journals (not the researchers who need to hang on at all cost as you note) to make things fair.
So, Steve, I take it you’re not planning on publishing in PNAS any time soon then? 😉
I had no idea about any of this. But I’ve always been suspicious of journals that require a paper to be “communicated” by a member, or editor, or similar noteworthy figure. That in itself smacks rather of your old boys network, even without these rather dodgy “contributed” and “direct submission” routes.
If I were planning such a submission, I’m sure that if the editors get wind of this blog my manuscript is dead in the water.
I did once try a regular-style submission a few years back–see my comment above to Steve Moss.
I’d like to state for the record that my PNAS paper went through the “I don’t know anyone important” normal peer review track 🙂
Ahhh, but with cracks as wide as those Steve mentioned, that’s inevitable, no?
(Ducks :P)
That’s what they all say…
You might be interested in the post ‘Does Reviewing Your Peers Create Better Results Than Peer-Review?’ by Phil Davis on The Scholarly Kitchen, and the article he mentions ‘Systematic Differences in Impact across Publication Tracks at PNAS’.
All the links seem to have gone from my comment just now – will repost!
You might be interested in the post Does Reviewing Your Peers Create Better Results Than Peer-Review? by Phil Davis on The Scholarly Kitchen, and the article he mentions Systematic Differences in Impact across Publication Tracks at PNAS.
Having trouble getting the links above to work correctly, so here are the URLs as back up:
http://scholarlykitchen.sspnet.org/2009/12/09/nas-member-peer-review/
http://scholarlykitchen.sspnet.org/
http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0008092 .
Irene,
Thank you very much for bringing the blog by Phil Davis to my attention; especially in the light of the technical issues you were having. I’ve had a chance to read through the blog (although not the original paper)–for those who don’t have time to to look this up, Mr. Davis is essentially saying that from an “impact factor perspective,” the direct submission mode of PNAS is mixed, with the top 10% of papers actually “outperforming” regular peer reviewed submissions to the same journal.
While as biologists we are seeking to put “metrics” on everything in order to mitigate any subjectivity, I think in this case it misses the point. For one, I’ve seen a lot of papers repeatedly cited not because of their good science, but because of the fact that their findings are wrong, inaccurate, or contradict everyone else’s–but nonetheless they need to be cited.
Second, many scientists will preferentially cite papers by more senior scientists, thinking (and in many cases for valid reasons) that these senior people will look more favorably on their manuscript if their own work is cited. And guess who “senior researchers” are? NAS members fall into that category. And they are the ones who can submit directly.
With regards to “a venue for new and groundbreaking work”–I simply don’t buy that. Yes, researchers may be conservative and skeptical–after all, that’s the way scientists are trained–but if the study shows the necessary rigor it should be accepted in peer review. Without peer review the whole system comes tumbling down and researchers will never know what they can build upon from the literature.
Pre-arranged editor submissions DO still go through board review, and they are peer reviewed by external reviewers. The pre-arranged editor is someone who agrees to handle the paper, that is all. Otherwise, its the same as submitting directly without a pre-arranged editor. I just did this, and that is the way it works. You have been misinformed.
Direct submission is the opposite of pre-arranged editor.
http://www.pnas.org/site/authors/editorialpolicies.xhtml