{"id":1909,"date":"2012-03-28T00:26:48","date_gmt":"2012-03-28T00:26:48","guid":{"rendered":"http:\/\/occamstypewriter.org\/vwxynot\/?p=1909"},"modified":"2012-03-28T17:06:20","modified_gmt":"2012-03-28T17:06:20","slug":"radical-reform-of-peer-review-and-research-support-or-rearranging-the-deckchairs-on-the-titanic","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/vwxynot\/2012\/03\/28\/radical-reform-of-peer-review-and-research-support-or-rearranging-the-deckchairs-on-the-titanic\/","title":{"rendered":"Radical reform of peer review and research support, or rearranging the deckchairs on the Titanic?"},"content":{"rendered":"

OR More on the proposed reforms at the CIHR, which I first blogged about last month<\/a> after taking my first look at the discussion document they published. I’ve since read the document in much greater detail, and also attended an informational \/ feedback forum held by CIHR at UBC, the university with which my research organisation is affiliated. I took notes at the forum (posted below) for the benefit of the PIs in my department – these notes should obviously be of interest to all CIHR-funded \/ fundable researchers, but I think there’s also lots of food for thought in there for non-Canadian readers, especially in the section relating to peer review rather than to the specific grant programmes that are being proposed.<\/span><\/strong><\/p>\n

Overall, my initial assessment stands that the changes to the peer review system sound very positive IF implemented as described; however, I think they’re going to have to work hard to convince the more sceptical PIs (of whom there were MANY at the forum). I also think that they’re going to need to put more thought into the specific eligibility and review criteria of the actual grant programmes, and a LOT more thought into how to handle the transition – either they haven’t spent enough time on this, or they’re being deliberately vague.<\/p>\n

If you don’t want to read the whole thing, here are my tweets from the event – read from the very bottom up to the very top to get them in the right order!<\/p>\n

\"CIHRforumtweets2\"<\/a><\/p>\n

\"cihrforumtweets1\"<\/a><\/p>\n

And here are the notes! Fingers crossed for Thursday’s federal budget…<\/p>\n

Background: current situation<\/strong><\/p>\n

The changes are apparently not written in stone yet \u2013 they\u2019re still gathering feedback. Several members of the audience seemed rather sceptical about this, claiming that the feedback process was just for show…<\/p>\n

In the transition from the old MRC (biomedical research only) to the current CIHR the mandate was broadened, but the CIHR does not currently meet that entire mandate. The system is stretched and needs to be made sustainable.<\/p>\n

The system is too complex: there are 53 standing peer review committees, whose mandates keep changing. 30-40% of all applications received by the CIHR don\u2019t truly fit into any of the existing committees and have to be \u201cforce-fit\u201d; this often results in grants being reviewed by two non-specialists.<\/p>\n

The CIHR used to fund high risk-high reward projects, but now funds mostly incremental science (there was some discussion about whether \u201cincremental science\u201d is a pejorative term: the CIHR claim that it isn\u2019t and that much of the science in question is of the highest standard).<\/p>\n

There is a continuum of funding levels among the ~3,000 PIs currently in the system (figure from the discussion document<\/a>):
\n\"CIHR<\/a><\/p>\n

 <\/p>\n

The average grant is around $162,000 per annum, but the range is huge and there is a long tail. Some PIs receive a single grant of $40,000 a year and do very well, whereas one particular PI has 12 grants totalling around $1 million per year.<\/p>\n

The CIHR aims to maintain the number of PIs supported through its open grants (~3,000), and maintain this range of funding as appropriate for the different types of research that fall into its mandate.<\/p>\n

Changes to peer review<\/strong><\/p>\n

The ultimate aim of the changes is to more quickly identify obvious yes \/ no funding decisions, thus allowing reviewers to spend more time on ranking the \u201cgrey zone\u201d grants in between.<\/p>\n

The biggest change would be that instead of standing committees, there would be a \u201ccollege of reviewers\u201d. All current CIHR-funded PIs, as well as all current reviewers, would be automatically included in a database with very detailed research, expertise, technology and patient population classification keywords. The best reviewers for each grant would be identified through this database by matching reviewers\u2019 keywords with those identified by the grant\u2019s PI. The CIHR Common CV and the keywords sections of the grant application forms are currently being revised to permit more robust self-identification of PIs and of individual grants based on these keywords.<\/p>\n

Since other countries with relatively small populations (e.g. Australia, NZ, France) have the same problems as Canada in terms of the difficulty of finding specialised reviewers who don\u2019t have a COI with the grant\u2019s PIs, the CIHR is in talks with its counterpart agencies in those countries to share its reviewer databases and allow any country to use any listed reviewer.<\/p>\n

Apparently \u201cthere really isn\u2019t a good peer reviewed literature on peer review\u201d, but based on the studies that have been done they would aim to assign 5-8 reviews per grant in order to more accurately rank the \u201cgrey zone\u201d grants. Each reviewer should get around 12 grants per round, but these would be shorter than the current Operating Grants \u2013 reviewers have been complaining about increasing levels of \u201cbloat\u201d in the number of figures, manuscripts, and other appendices that don\u2019t currently count towards the page limit.<\/p>\n

They aim to reduce reviewer burden by moving to multi-stage reviews. The actual number, review criteria, and timeline of these stages would be different for programme vs. project grants, but both streams would involve some form of online discussion before moving to face-to-face meetings for the final stage of discussion and funding decisions. NB ~800\/2300 application to the last round scored worse than 3.5 – they\u2019d like to remove such \u201cno-hoper\u201d grants from both streams at the 2-3 page proposal stage. They\u2019ve had lots of complaints about reviewers\u2019 current travel burdens, and therefore plan to have fewer face-to-face meetings than under the current system. They also explicitly stated that any online system would absolutely have to<\/em> permit reviewers to argue with each other as well as just upload their own reviews J They may<\/em> also allow reviewers to ask applicants for clarification of specific aspects of the grant.<\/p>\n

A grant could have different reviewers assigned at each stage of the multi-stage process, and at resubmission \/ renewal compared to the original submission. They acknowledged that the latter scenario in particular is not ideal and that all funding agencies struggle with this, but emphasised that funding decisions are always about rankings within each round. <\/em>They claim that at the NIH, where resubmissions are supposed to go to the same reviewers as the original application, \u201ceveryone knows that they usually ignore the original reviews anyway\u201d.<\/p>\n

They said they would like to find a way of rewarding reviewers. This generated much discussion. One suggestion that reviewers, perhaps especially those who fly from Vancouver to Ottawa and back every six months, should be given automatic extensions to their own grants. This caused a round of applause (almost a standing ovation, actually), but apparently the CIHR can\u2019t afford it.<\/p>\n

Reviews will be more structured than they are now. Apparently some people write 6 page reviews of every grant assigned to them, but the average is 2 pages.<\/p>\n

They would like to implement specific training in peer review for new reviewers \u2013 online courses, mock review sessions with real grant applications, etc. Some individual departments \/ faculties already do this, but the CIHR is not currently a partner and would like to adopt best practices from around the world.<\/p>\n

There was a suggestion from the audience that there should be an arms-length budgetary committee that meets after the rankings are finalised, to assess the impact of grant budget cuts on each team\u2019s ability to carry out the proposed research \u2013budgets are currently cut by at least 20% across the board, but for some projects (e.g. clinical trials and associated translational studies) this would mean that the project could essentially not go ahead. The CIHR promised to include this suggestion in their internal discussions.<\/p>\n

The new programmes<\/strong><\/p>\n

General<\/span><\/p>\n

Much scepticism was expressed at the notion that the CIHR is not implementing a two-tier system.<\/p>\n

NB the amounts listed in the document published on the CIHR website (Programme: $300k per year for 7 years; Project: $125k per year for 3-5 years) are NOT going to be hard caps. There will not be any career limits on the number of grants in either stream for which a single PI can be awarded.<\/p>\n

In both streams, a certain percentage of the available funds will be set aside for new investigators, whose applications will compete against each other rather than in the general pool. The definition of \u201cnew investigator\u201d in the document they published is \u201can applicant who has either never applied before to CIHR, or whose last degree ended five years or less before the original competition date\u201d, but this is still being discussed and the definition may well change.<\/p>\n

Teams of PIs, as well as individuals, will be able to apply for both types of grants.<\/p>\n

The CIHR is in discussions with major institutions about how to structure the \u201cinstitutional support\u201d aspect of the review criteria, but emphasise that this new section will absolutely not result in any loss of academic freedom. They also stated that this section is not about matching funds, but rather ensuring that all funded investigators have the required facilities \/ support \/ capacity to carry out the programme or project.<\/p>\n

Programme grants (7 years)<\/span><\/p>\n

There will probably be one competition per year.<\/p>\n

They see this stream as being for people who are \u201cconsistently successful\u201d under the current system and want longer term support and less frequent grant applications \/ renewals. Track record would be a major component of the review. However, there are no \u201ccurrent CIHR support\u201d eligibility criteria for this stream \u2013 i.e. you don\u2019t have to have held CIHR funding before to apply to this stream, and there is no upper number of grants currently held that would prohibit you from applying.<\/p>\n

You can only hold one programme grant at a time. PIs who are awarded programme grants will not be able to apply for regular project grants as nominated PI, but might<\/em> be able to be co-applicants on other PIs\u2019 project grants (to be decided), and will definitely be able to apply to selected special \/ strategic initiatives.<\/p>\n

They have been talking about a target renewal rate of 50% for these grants, but have heard LOTS of complaints about the impact of such sudden and massive cut-offs for the unlucky half of applicants, and are therefore giving more thought to this target.<\/p>\n

Project grants (analogous to the current Operating Grants)<\/span><\/p>\n

There will probably be two competitions per year. However, they may have to change this as they would anticipate a massive spike in project grant applications in the first round after the programme grant decisions are handed out.<\/p>\n

Transition<\/span><\/p>\n

There will be at least one full year\u2019s notice of the implementation of the new programmes; they expect to provide the first funds under the new system in 2014-2015.<\/p>\n

They haven\u2019t made any firm decisions yet about how the transitional period will work. They will run pilot schemes before making the full switch, and make sure that the needs of people at all stages of the career pipeline are being met \u2013 most expressions of concern via the feedback form have been from mid-career researchers (however, most complaints on the day were made by the older members of the audience, from what I saw). They say they will closely monitor the impact of the changes, and tweak the system as needed during the first few years of the new structure.<\/p>\n

People with multiple funded current grants might <\/em>be offered a \u201csmooth transition\u201d to the programme scheme, whatever that means (it was left very vague).<\/p>\n

Overall CIHR Budget<\/strong><\/p>\n

The atmosphere got gradually tenser throughout the session \u2013 you could sense the frustration by the end. Someone stood up and loudly proclaimed that the CIHR are \u201cre-arranging deck chairs on the Titanic\u201d and that the only solution to the current problems is more money. Several people yelled \u201chear hear\u201d and things became rather heated.<\/p>\n

The CIHR do not know how they\u2019ll be affected by Thursday\u2019s federal budget, and a question from the audience about \u201cwhether Mr. Harper\u2019s intentions toward the CIHR \u2013 budgetary and otherwise \u2013 are honourable\u201d was side-stepped…<\/p>\n

Stay tuned!<\/span><\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

OR More on the proposed reforms at the CIHR, which I first blogged about last month after taking my first look at the discussion document they published. I’ve since read the document in much greater detail, and also attended an … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45,7,48,43,3],"tags":[],"class_list":["post-1909","post","type-post","status-publish","format-standard","hentry","category-canada","category-career","category-grant-wrangling","category-politics","category-science"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/posts\/1909","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/comments?post=1909"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/posts\/1909\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/media?parent=1909"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/categories?post=1909"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/vwxynot\/wp-json\/wp\/v2\/tags?post=1909"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}