I was interested by an article in this week’s Nature1 about the Italian government outsourcing some peer review to the American NIH.
This seems like it could be a useful exercise for some governments, in some circumstances. However, I was surprised to read that Italian researchers will not be brought into the peer review committees. It seems to me that there would be more potential for two-way learning if Italian science were represented during the entire process. I read a lot of blog posts about the NIH (and NSF) systems, and there are clear cultural differences even to Canadian peer review, so one might expect that there would be an even greater gulf between US and Italian practices. Every country will have different research priorities and academic structures, after all, and proposal rankings decided by foreign scientists might not translate into the optimal research portfolio for a smaller state.
It will be interesting to see how this experiment pans out, but I expect that wholesale outsourcing of peer review to a foreign government will always be an exception rather than a rule. Mixed nationality peer review committees, on the other hand, sound like a very interesting idea. I’m sure this already happens within the EU, and with specific collaborative agreements between different countries; does anyone have any experience in this area?
1 accessed through the website, as all attempts to click RSS feed items from within Google Reader are resulting in “DOI not found” errors from The DOI System, rather than taking me to the relevant article on the Nature site in the usual way.
Hmmm, interesting experiment. It’s not entirely clear what they are trying to avoid by outsourcing. Most smaller research communities suffer from conflicts of interest in peer review at some point. Can we read from this that there is a problem with scientific “nepotism” in Italy?
bq. Mixed nationality peer review committees, on the other hand, sound like a very interesting idea.
It happens in Finland. They even fly Canadians over to be on the committee. It makes sense: the community is so small that everyone knows each other, and for any targeted calls, everyone with the necessary expertise would be applying anyway.
bq. Can we read from this that there is a problem with scientific “nepotism” in Italy?
Darren, I think you can drop the word “scientific” from that sentence and it will still hold true.
I think Cath’s main point ties in with this post in some respects. I think “outsourcing” can be a good idea in both grant application and paper reviews.
Asking fresh eyes to view a project can give really useful insight, from within and between fields.
I can understand using reviewers from abroad. The Austrian funding agency does this, because Austria is so small that it might otherwise be difficult to find reviewers that are knowledgeable in the grant’s subsubsubfield but not in a conflict of interest. Italy is a lot bigger, of course, but its scientific community might still be a bit small in some areas. And ‘outsourcing’ the whole process is yet another level.
The UK funders already draw many of their reviewers from outside the UK, and have done for some time. So I think getting reviewers from outside your own country is already almost universal. And the international journals I know, even if run by largely national scientific societies, have fully international editorial boards and review panels.
I suppose it is harder to produce an international reviewer pool if your grants are submitted in a non-English native language, but I would have automatically presumed the Deutsche Forschungsgemeinschaft (for instance) would still make a lot of use of German researchers in the “German diaspora”, esp. in the US. The article actually says the DFG use 22% of non-Germany based reviewers. I would be interested to know comparable figures for other European countries’ funding agencies.
The more interesting point here is it suggests the Italians are outsourcing the entire PROCESS, not simply trying to use reviewers outside Italy. That does tend to suggest systemic problems with the entire system for handing out the money.
Here’s a relevant quote from the story:
“The US National Institutes of Health (NIH) is gearing up to begin a review of about 1,000 biomedical research grant applications for the Italian government, an experimental collaboration that comes at an inconvenient time for the US funding agency…….”We took on this project before the Recovery Act was passed, and we never would have taken on the Italian applications if we had known what our workload would be now,” says Antonio Scarpa, director of the NIH’s Center for Scientific Review. “Nonetheless, we are honoured to assist the Italians.” It is the first time the NIH has provided systematic technical support for another country’s grant applications, he says.”
Cath – the Nature table of contents RSS feed worked for me last night – I test it each week on a Wed eve in my reader. Sorry that it was glitchy from your end. Hope OK now.
Maxine, Cath: I often found that the DOIs in Nature’s RSS feed didn’t work when they were first published. Seeing as Crossref are damned near instantaneous I wonder if there’s a delay between publishing and the DOIs being submitted to Crossref?
Thanks everyone for this great discussion! It’s great to hear from people in different countries – the internet rocks.
the community is so small that everyone knows each other, and for any targeted calls, everyone with the necessary expertise would be applying anyway.Austria is so small that it might otherwise be difficult to find reviewers that are knowledgeable in the grant's subsubsubfield but not in a conflict of interest.Absolutely. This is actually one of the major differences between an NIH R01 grant and the Canadian (CIHR) equivalent – we write ours at a different level of detail because the pool of reviewers is smaller and will not usually contain an expert in the subsubfield.
It just seems counterproductive to actually exclude reviewers from the country that is funding the research!
the international journals I know, even if run by largely national scientific societies, have fully international editorial boards and review panels.Indeed – but the criteria for publication of quality science
should beare the same across international boundaries, whereas funding decisions are usually made with additional, state-specific criteria other than just the quality of the science. It is not clear how well the rankings decided by solely American reviewers fit the research priorities of Italy, for instance. Then again, maybe this is a good thing that reduces the impact of political interference!I suppose it is harder to produce an international reviewer pool if your grants are submitted in a non-English native languageHeh – Canadian government funding agencies allow submissions in English or French. Choosing French narrows the pool of subfield experts even further. Of course, no-one would ever do this on purpose in order to practically guarantee the utilisation or exclusion of a specific reviewer. (Thankfully I have never been asked to work on a submission in French myself – I’ve never even seen one go in from this institute. But one hears certain rumours when one attends grant writing workshops).
The RSS feed links are working this morning, but I did check a few times yesterday, and it must have taken at least 4-5 hours. Other NPG feeds were working fine during this time.
I often found that the DOIs in Nature’s RSS feed didn’t work when they were first published
Not just Nature. I have had probkems with DOIs from other weeklies when I try to access them on the day of publication. It is a big nuisance.
Frank/Richard – agreed. We picked up on this delay fairly recently, and now release our dois a little earlier than the journal (not sure of the exact time lag, maybe a few hours). You are right that there is a lag of connectivitity somewhere along the line between the various services if you release the dois simultaneously with the actual content. As Frank says, its to do with the way all these services connect up and set themselves, or the Internet itself. The only way that has occurred to me to deal with it is to send out the dois early, but let me know if you have other ideas. I hope that Nature is now OK. Please let me know if you experience the problem again for the Nature table of contents in RSS – with the relevant issue and the timings, so I can let our technical teams know. Much appreciated, thanks.
Score!
Thanks Maxine. From my own experience with entering DOI lookups it seems to happen very very quickly indeed, but maybe the gerbils get run-down?
Hello again, I have spoken to our technical team and they send the dois for Nature at 3 p.m. on a Wednesday. The issue and associated table of contents alerts (email and RSS) go out at 6 p.m. (or a bit later if anything has been held up in the journal production process). So there should not usually be any reason for the doi links to be dead. However, we only began this system 3 or 4 months ago, which is when we realised that the problem was occurring with simultaneous release.
Last week, there was an IT emergency at NPG which affected the entire web production system, so the team had to find a workaroud to get the journal out. This has now been fixed, but it could account for the problem you had, Cath, with last week’s issue specifically.
However, there is something odd about the whole thing because the news table of contents RSS feed goes by URL not doi (in my reader, anyway), so you nobody should have the doi problem concerning Nature news (the other Nature article all use the doi system in their RSS tables of contents).
My home keyboard is dying on me and skipping letters: I did not check preview before posting, sorry. Last part should read:
so nobody should have the doi problem concerning Nature news (the other Nature articles all use the doi system in their RSS tables of contents).
Thanks for looking into this Maxine! I’ll let you know if I see the same problem again, but it does sound like it might have been limited to just this week’s issue.
However, we only began this system 3 or 4 months ago, which is when I left Australia.
Ah…
I’m not too familiar with the peer review process, but this has sparked my interest. thanks, Cath!
Most of the above is right. Italy has a rather small scientific community. There is not adequate specific expertise for innovative research. There is also a high risk of conflicts of interest. The initial review process will be done by expert referees from USA, then the evaluation will come back to a national committee that will review discrepancies among reviewers and justify possible limited changes in ranking, then the process will be audited.
It is like in a production line: those that produce must know exactly their portion, and are controlled by those that do the quality control, those that do the quality assurance must know the process, check on it and control the entire process. Why are we doing it? because this is the best method. Why with all foreigners? To set the standards and find out without bias what is the level of our current research. We will then initiate an international collaboration with other countries to exhange and mix reviewers, but the first try was to be all international to start. The most important thing righ now is to establish that merit comes first and that “priorities” come next. More then else this is a message of change. Finally all the evaluations and revisions of ranking will go online. We are the first ones to be tired of nepotism.
Trisha, there’s a lifetime of study material out there, so be careful! 😉
Giovanni, thank you so much for your input. It’s great to hear the context of the entire process that you’re planning, it sounds extremely interesting. I hope we’ll get to hear the results of this intiative!
I tend to agree with Giovanni Lucignani’s position, including his final statement about being so very tired about nepotism. Outsourcing in this case was probably thought as the safest way to avoid conflicts of interest.
I want to thank Maxine for pointing my attention to this discussion on Cath’s blog. I had overlooked it when I had prepared a forum topic on the Nature Network Italy group (as well as a blog post). Those project proposals that will be reviewed this year at NIH were indeed written in English, and were designed for will- be, fresh PIs in their mid 30’s. Applications are much shorter than the typical NIH R01s.
Thanks, Massimo! I am going to check out your Nature Network Italy forum post when I get a minute, this week seems to have been more busy than usual, even for me!