Indigestible Committee Paperwork

Each summer it is standard for publications to produce lists of exam howlers to remind us just how woefully ignorant some of our students are at all levels. I have never seen a list of comparable statements regarding crass errors produced in grant applications, but upon occasion referees do not mince their words.  Having spent much of the weekend wading through a pile of (metaphorical) paper – it is of course all electronic material now – consisting of proposals and referees’ comments, I was amused by the following two comments which leapt out of the page at me:

‘Within the case for support, the term “paradigm shift” is used (misused) twice and “step change” mentioned three times, yet it is never clear what could possibly be considered revolutionary about the proposed work. ….’

‘This is a badly written, ill-thought out and poorly developed research proposal that contains a number of serious flaws and is not firmly based on current knowledge of [the field].’

Very few referees comment specifically on the mandatory (for UK research councils) ‘pathways to impact’ statement but, given how little concrete detail of what is wanted and/or required is out there on their web pages, I thought I would proffer some personal thoughts on what not to write, and possibly the odd pointer for more positive statements that could be made by the PI (this is a fairly UK-centric post, I should say).  These remarks – like those I made in a previous post about committee membership – should be taken as composite examples of the sorts of things that can go wrong, and which may stick in the committee’s minds for unfortunate reasons. I will not put any literal quotes from impact statements I have read in what follows, but the general messages will be along the lines of those I have seen, and reflecting the nature of the particular grant-giving committee I am most familiar with currently.

Many people believe the Pathways to Impact statement is an opportunity to hype their field – just the place to discuss a paradigm shift (as above) in an entirely vague and meaningless sort of way. Whatever it is they are going to do, it will solve all the ills of mankind, revolutionise the production of something or other and allow us to fly to the moon. This is not really helpful. My impression is a bit of quantification is particularly valuable – we are scientists after all – so if a new drug is going to be developed what is the size of the potential market in the UK/ the world?  If the project is to solve some existing bottleneck, how big is the problem? If a new healthcare technology is being proposed, enumerate the number of sufferers who potentially could benefit. Surprisingly often a committee member will want to know about the economics, if a PI is claiming to be going to develop a potential new product, to see if it is ever going to be financially viable rather than just esoterically clever.

Turning to outreach, if the team claim they are going to visit schools as part of their activities, why not set an explicit target eg. 3 schools a year, or engage with a science festival twice during the lifetime of the grant.  Most people inevitably stress what they have done in the past –  perhaps they talked to a local newspaper in 2001 – whereas what strikes me as preferable is to stress what will be done in the future and ideally how it connects to the specific proposal, and that is why targets or milestones are desirable. Although it is impressive to be able to say that members of a team have presented their results to members of Parliament several times, how will they ensure they do so again?

And really it is little good saying that in the past the PI has taken out 2 patents and worked with several companies unless it is clear how patents,  licensing and industrial collaboration might work in the future. What has happened in the past is clearly relevant as demonstrating, perhaps, an entrepreneurial mindset, but it is only relevant if the future trajectory appears to have been considered too. So can something more specific be said than that a department is an ‘industry-facing department’ (horrid phrase,  but one that did turn up in a case I read); or that it might have had, in a vague kind of way, a long track record of industrial collaboration.  Rather like my experience on the REF pilot panel when such vagueness was also frequently present, such statements are really not explicit enough to judge anything beyond a cuddly feeling that maybe the particular department has had industrialists set foot on the premises. Why not say something about what sort of industrial contribution (cash or in kind) has flowed into the department in the past, how much of it in the direction of the particular PI and what their expectations might be over the next 3 years? And to say that attempts will be made to ‘upscale production’ in the future, if the grant is successful, without explaining a mechanism and an outline indication of what success – including on financial grounds – might look like, is just a bit too fuzzy

Whatever my own personal views about the motivation that has led to the introduction of these Pathways to Impact statements and the precise form in which we are supposed to describe our activities, it does us as scientists no harm to reflect on why we are doing what we are doing. We do not have any sort of intrinsic right to do science for the fun of it, but equally some science is inevitably going to be  less immediately translatable into economic worth than other projects. But we should anyhow all be thinking about what we can do to excite youngsters and the public, as well as train the personnel we will employ on grants in a variety of transferable skills, including outreach.  It shouldn’t be difficult to define these outcomes. If you are about to write one of these statements do look ahead, not just quote the past however wonderful your research and activities may have been previously; do be concrete and not vague and full of hyperbole.  You should consider that I and my colleagues will have read dozens of these statements over a rather short period of time in the run-up to the committee meeting,  so have some thought that just maybe my eyes glaze over when faced with yet another page of statements along the lines of

This proposal aims to develop new functionalities of [insert generic technology here] to support the next generation [insert protocol here] which will transform the production of [insert your favourite molecule here].   We will concentrate, just as we always have, on writing lots of peer-reviewed papers and travelling the world to exotic places to talk to our friends at high level conferences.

I’m afraid that doesn’t cut the mustard with me.

This entry was posted in Communicating Science, Research, Science Funding and tagged , , , , . Bookmark the permalink.

16 Responses to Indigestible Committee Paperwork

  1. Heather says:

    Oh, Athena, how very very true. Again. Sigh.

    Perhaps what you are asking for is an explicit commitment, if a person has gone to the trouble of outlining their outreach activities in the past, to reproduce something like those in quality or frequency in the future? That shouldn’t be too difficult. While I have no idea how many junior high school students will apply to do their one-week shadowing session in my lab next year, I can say that I have welcomed four last year and two, the year before that, and consider it likely that I will receive a few next year as well.

    I must give you some love for using the word “proffer” so smoothly.

    Ack! We have captchas now!

  2. Heather says:

    Ack. But no way to go back and correct inept comments. Next time I will try to type your name correctly – my fingers were faster than my brain. Apologies.

  3. Stephen says:

    Athene – This post has triggered a few thoughts – not all of which I can articulate right now. I feel your pain as a committee member wading through such tedious paperwork.

    But part of the problem I feel is the mis-match between what many scientists see themselves as doing and the impact game that they are obliged to play by the Research Councils. Many are probably a bit bent out of shape, feeling they have to concoct some sort of impact assessment, even if their research is fundamental with no clearly envisaged economic impact. I know the RC’s say that other forms of impact are permissible but I bet there is widespread suspicion that economic impact may be the only form that really counts. Hence the vacuous hyperbole…

    • peeceedee says:

      Well said.

      • I think it is really important we don’t simply see impact as an economic issue. This was misplayed early on and must be resisted. As scientists we have impact in many different ways, and we should be able to articulate that in these plans. When it comes to the REF, impact will mean many things to different communities and I think our humanities friends seem to have got this worked out better than us now. It was noticeable in the report on the REF pilot that the English panel (if I remember correctly) seemed more enthusiastic than Physics!

  4. cromercrox says:

    Most instructive. We at your favourite weekly professional science journal beginning with N always get suspicious when a manuscript concerning some interesting curlicue on an outlying gargoyle perched on a minor flying buttress on the chapter house next door to the Cathedral of Knowledge is touted as having ‘therapeutic implications’.

  5. Hi, Athene.

    I discovered your blog only recently via the pages of “Physics World” – you’re featured in the “Web Life” section this month. I therefore hope you’ll excuse me for commenting on this post rather late in the day.

    While reading many of your other posts, I found myself nodding along in agreement. But not this one. It saddens me that a scientist as well-respected and influential as Athene Donald should castigate colleagues for not taking the entirely nonsensical ‘Pathways to Impact’ process seriously. As you may know, I no longer review (and therefore no longer submit) EPSRC – or, indeed, RCUK – proposals due to the imposition of the impact statement requirement. This means that when my current EPSRC fellowship runs its course (~ Oct.2013), I will need to very carefully consider my career options. I feel *that* strongly about the farcical ‘impact agenda’. I’ll try to explain just why as dispassionately and concisely as possible in the following (although ‘dispassionate’ and ‘concise’ have never been easy for me…).

    The reason that you read so many “vague”, and “meaningless” impact statements from your peers is spelt out rather well by Stephen in his comment above. Scientists indeed concoct vacuous hyperbole so as to tick the ‘impact box’. The vast majority of academics I know (spanning the physical, life, and social sciences) see the pathways to impact statement simply as an irritation. It’s two pages of nonsense that they are obliged to complete.

    This is not just my (or Stephen’s) perception of academics’ response to RCUK’s impact agenda. For example, during the Q&A session following a presentation by members of the REF Impact Pilot for Physics at the IoP late last year (more on this below), a physicist from Cambridge pointed out that the “only people he knew who agreed with the HEFCE/RCUK impact agenda” were, errrmm, in HEFCE or RCUK. That’s a pithy summary of academics’ response to the ‘pathways to impact’ scheme.

    But it’s much, much more than *just* an irritation. The RCUK ‘Pathways to Impact’ policy – regardless of whatever sophistry they use to claim otherwise – represents a sea change in how UK science is funded. This is no better demonstrated than in tip #1 from RCUK’s “Top Ten Tips on How to Complete the ‘Pathways to Impact’ Statement”::

    “Draft the Impact Summary very early in your preparation, so that it informs the design of your research”

    So, identify your ‘users’ and design your research project accordingly. That’s not science. It’s the ‘D’ part of “R & D”.

    Perhaps the most dispiriting aspect of the debates/discussions with RCUK/HEFCE in which I’ve been involved was having to explain to a senior RCUK representative just what the term “disinterested science” means. One might ask why Robert Merton included “disinterestedness” as one of the key norms of science. I am not for one moment suggesting that all science routinely conforms to Merton’s norms – a perhaps rose-tinted view of the social interactions which underpin science, or ‘the Legend’ as John Ziman would have it – but, nonetheless, shouldn’t we at least aim to ensure that disinterestedness is a value embedded at the heart of the scientific method?

    As regards the “socioeconomic” vs “economic” impact question, let’s first once again tease apart the RCUK/HEFCE sophistry. David Delpy, CEO of EPSRC and the RCUK ‘Impact Champion’, stated in 2009 that what he wanted to see was a “culture change” in academa, so as to “shorten the innovation chain”. The ‘pathways to impact’ process is fundamentally about making scientific research more responsive to the private sector and the market (as spelt out by Warry, Sainsbury, the ‘Next Steps’ innovation framework etc..etc…). Let’s not fool ourselves otherwise. John Armitt hit the nail on the head regarding RCUK priorities back in 2008 when he said “one man’s economic impact is another man’s societal impact”.

    I’m a scientist. Not an engineer. Not a technologist. And certainly not an entrepreneur. I did not take up a position as an academic scientist to do near-market research which has demonstrable short-term socioeconomic impact. In the eyes of RCUK and HEFCE I am therefore not doing my job correctly. This rankles. A lot.

    The other RCUK/HEFCE counter-argument that really gets my goat is the suggestion that those who oppose the ‘pathways to impact’ nonsense are simply ivory tower academics who don’t want to be accountable to the taxpayer. I’m a publicly-funded academic. As I see it, I have an *obligation* to get involved with outreach and public engagement activities (and, indeed, spend a not-insignificant amount of my time on these aspects of my job, because, fundamentally, I enjoy them.). I do not, however, have an obligation to do near-market research that can be commercially exploited. Nor, and this is a key point, is there a direct link between my public engagement activities and the quality of a research proposal I submit.

    I’ll leave you with a rather telling example of the flaws at the heart of the impact agenda. At the meeting re. the REF Impact Pilot panel results, mentioned above, Bill Wakeham seriously suggested that number of column inches in the local/national/international media could be used as a metric for impact. But not before those numbers were normalised so as to account for the intrinsic ‘sexiness’ (his term, not mine) of certain areas. So, for example, because astronomy is so much more ‘publically engaging’ than some other areas of science, those involved in public engagement with astronomy need to have their column inches normalised ‘down’.

    When we’ve reached this level of ‘analysis’, we’ve moved from the flawed to the farcical.

    Philip

  6. Apologies. Despite my stated aims, the last comment was neither concise nor dispassionate!

    Philip

    • Philip, your strong feelings on this subject are well known, but I do feel you are taking an extreme view of what the Impact Statement is all about (or meant to be). It started off being badged as Economic Impact, but it really isn’t meant to be that now, hence its title of Pathways to Impact. Many, if not the vast majority, of the statements I see (as it happens I am looking at BBSRC statements not EPSRC, but in principle that shouldn’t alter their tenor), are not about spinouts or near market research. However, iit is not unreasonable for PI’s to consider how they will be training of their staff (something BBSRC are keen to give more prominence to), outreach and public engagement, things you say you do and I’m sure take very seriously. Getting hung up on how close to market a piece of research is and trying to sell it in this statement are, I believe, missing the point. I said this in my response to Stephen and I strongly believe it. And, within that framework, it is not necessary to write the kind of waffle I was referring to, or only looking to the past. Much more worrying in my view is the change in language in the EPSRC Delivery Plan to becoming ‘sponsors’ of research. That really does imply a change of emphasis that will cause problems for the community.

      My views on the REF are less positive, having sat on the Pilot Panel, because although it is not a requirement that all entered staff have to have something with economic worth, the more who do the easier the submission will be. Additionally it was very difficult, at least in the way the pilot questions were drafted, to assess the cases so quantification was definitely only with large error bars. I hope the full-blown version will have more useful formulation of the questions so that it is easier to separate out the good cases from bad, and not be misled (once again) by hype. In particular, the public engagement angle was hard to quantify; is it enough to have talked to schoolchildren/had column inches , or do you need to be able to show some demonstrable outcome? Having done the RAE and the REF pilot, though, I will not be facing up to this challenge again!

      • Athene,

        A brief(-ish) response for now – I’ll get back to you later tonight on the various other important points in your comment.

        The movement of ‘EPSRC’ to become ‘sponsors’ of research is part of precisely the same ‘continuum’ as the impact agenda. EPSRC/RCUK didn’t introduce an assessment of impact because they suddenly became concerned that academics weren’t doing enough public engagement activity. They introduced an assessment of impact so as to show to BIS/the Treasury/Government that they were taking seriously the recommendations in a variety of reports that academia became more ‘business facing’ or, preferably, ‘business-led’.

        Let’s remember that the first item on the RCUK list describing what the research councils mean by “impact” is as follows:

        “fostering global economic performance, and specifically the economic competitiveness of the United Kingdom”

        And, yes, I am well aware that there are other categories. Number two on the list is:

        “increasing the effectiveness of public services and policy, ”

        “Increasing the effectiveness” is an interesting turn of phrase! Are you aware of the changes that are currently happening to ESRC which mirror EPSRC’s drive to rebrand itself as a “sponsor” of research?

        The re-badging of “Economic Impact” as “Pathways to Impact” is exactly that – a ‘re-badging’. The motivation for the introduction of the ‘pathways to impact’ process was overwhelmingly – I’d go so far as to suggest solely – economic. If you have evidence to the contrary, I’d very much like to see it.

        More later.

        Philip

        P.S. …and that should have been “publicly” rather than “publically” in my previous comment, of course. Tsk. Academic standards at Russell Group universities aren’t what they were…

  7. Athene,

    I didn’t want to upload yet another tortuously long comment so I’ve embedded links to a number of papers in the following. These provide a rather more nuanced description of my opposition to the impact agenda.

    I think that the central issue of contention is as follows. You (and RCUK) are of the opinion that the ‘Pathways to Impact’ process enhances the public value of academic research. I believe that RCUK’s focus on ‘user engagement’, perhaps counter-intuitively, will weaken the societal value of publicly-funded research. The reasons are described in “Reclaiming Academia from Post Academia” [Nature Nanotech. 3 60 (2008)] and in “Public Science: A Public Good?” [Nanotech. Perceptions 4 101 (2008)].**

    I don’t do science to engage with users. Nor is the motivation for my research the amount of impact it will have. This second point is apparently very difficult for those at EPSRC/RCUK to grasp. David Delpy, in particular, seems to be of the opinion that scientists are entirely motivated by a need for their work to have impact.

    Perhaps I’m highly unusual in stating that a consideration of impact plays absolutely no role in my choice of research topics? I don’t think so. I don’t know if you saw the episode of BBC4’s “Beautiful Minds” last year which featured Tim Hunt? If not, it’s available here . It’s an inspiring and moving description of just why science must have at its heart the freedom to explore, to be creative, and to extend the limits of what we know (…without a monthly focus on whether we’ve satisified the user requirements in our ‘Pathways to Impact’ statement and/or completed deliverable 2.7.1A/0FCC in the project Gannt chart).

    Philip

    ** I also had an enjoyable ‘spat’ with Terence Kealey (VC of the University of Buckingham, as you no doubt know) last year on the subject of whether scientific research is a public or a private good (or, as Terence would have it, an invisible college good). Although Terence and I have diametrically opposed political views, he – unlike those at RCUK who are driving university research from a public to a private good – is willing to engage in debate.

  8. Philip, this isn’t the place for a long discussion of your points. You have indeed become a champion for the ‘don’t believe in impact as defined by RCUK’ view and we will never agree. My original post, however, was not meant to be a discussion of the pros and cons of having such a thing as a Pathways to Impact statement, simply a discussion of how badly people accomplished the task of writing one.

    I may not ‘do’ science to engage with users, but I certainly know I wouldn’t do any science that I thought had no useful end point. It was one of the key reasons, as a teenager, I turned my back on the interesting but (as I saw it then) irrelevant world of cosmology. I like doing stuff that has some motivation beyond being clever, though (typically) the Guardian completely overhypes and gets wrong the kind of stuff I do do when it says I work on things ‘such as revolutionary treatments for Alzheimer’s.’

    I think to think that the choice of one’s research is, or should be, influenced by impact is completely to miss the point. Having chosen your research field RCUK want you to state how it might lead to impact, which it does no harm to think about. And I absolutely do translate impact in a very broad sense myself, when I read the statements submitted. You have chosen to take a very extreme position, I (and many like me) are more pragmatic. The real problem lies, of course, with the Treasury who – much more than the Research Councils – can’t think beyond numbers, metrics and pure cash.

    • And, finally, on the issue of the ‘relevance’ of certain aspects of science (as you raise the question of the ‘usefulness’ of cosmology in your comment above).

      I found it intriguing that in another blog post you wrote that you were somewhat frustrated by what you saw as the popular conception that only cosmology/astronomy/’big science’ could be used to inspire the public about physics. That’s not how I see the issue at all.

      A high proportion of the GCSE and A2/A-level science curricula is focussed on the ‘real world relevance’ of the science, at the expense of what I would argue are the more inspirational aspects. A very large number of students who take up physics at university are inspired by what might be described as the more ‘esoteric’ aspects of the subject – quantum mechanics, particle physics, the origin of the universe etc…

      This is linked to your point that “I may not ‘do’ science to engage with users, but I certainly know I wouldn’t do any science that I thought had no useful end point.”.

      O.K. – define “useful end point”.

      – Did Einstein’s general theory of relativity need to be exploited in GPS technology before it was useful? (It’s not enough that it revolutionised our understanding of the universe?).

      – Was Watson, Crick, and Franklin’s work on DNA not ‘useful’ until it was exploited in e.g. biotechnology and genetic engineering?

      – How about the search for dark matter/dark energy? Only ‘useful’ because of the technological spin-offs or might it just possibly have a societal value without its commercial exploitation?

      ….

      Philip

      • I have never said that I didn’t approve of blue skies research. I merely said I personally wouldn’t do any science that I thought had no useful end point. That is my personal philosophy about what I choose to do and not how I judge other people’s research. You shouldn’t extrapolate too far. And as far as I am concerned, had I been around at the time of Crick and Watson it would have struck me as immediately obvious that understanding the genetic code was a wonderful thing to do and the endpoint was obvious. That’s nothing to do with genetic engineering or biotechnology or anything else derived from their work 50 years later with or without an economic value.

        As for my comments about the wow factor, again they are my personal views about what excites me. What frustrates me about this issue is the way science (in this case more specifically physics) is presented in the media. The LHC and black holes are seen as sexy, food (say, something I am sensitive about) is just funny if physicists study it. And that has nothing to do with pathways to impact, I felt like this long before anyone had invented that phrase. The school syllabus does have a fair amount of the everyday material world, and I’m pleased to see it. But Horizon does not, nor has Brian Cox yet done a programme about the physics of the stuff around us in our homes etc. Tony Ryan’s Royal Institution lectures a few years back were an exception.

  9. “You have chosen to take a very extreme position, I (and many like me) are more pragmatic”

    Indeed. And it is the high level of pragmatism – or the “playing the game” aspect – which allows RCUK/HEFCE to implement changes to which a very high percentage of the research community are directly opposed. Whatever happened to the academic role of ‘speaking truth to power’?

    My ‘extreme’ position, as you put it, arises because RCUK and HEFCE are distorting – I would go so far as to say perverting – the ethos of academic research via the impact agenda. Have you read Jennifer Washburn’s “Universities Inc: The Corporate Corruption of Higher Education” or Greenberg’s “Science for Sale”? These deal with the US situation but the parallels with the UK system grow ever stronger. I would also recommend the “Scientists for Global Responsibility” report on corporate influence on UK academic research – “Science and the Corporate Agenda”.

    You can dismiss these concerns as being extreme, if you like. I would argue that it is naive (in the extreme) to blithely accept RCUK arguments that the impact agenda is solely driven by the public interest. It’s fundamentally a re-structuring of university research to make academia more responsive to the needs of business and industry. The impact agenda was the first step. EPSRC’s announcement that it will now act as a research ‘sponsor’ is but the next logical step in the process (getting rid of those pesky project studentships that are difficult to align with strategic interests along the way – a canny move).

    I also don’t believe that all the blame can be laid at the feet of the Treasury. The dynamic is much more subtle than that. There are those at EPSRC/RCUK who, for those very reasons of ‘pragmatism’ you highlight, have invested a significant amount of their reputation in the impact agenda. In addition to pressure from the Treasury, there is certainly not an unwillingness in the upper echelons of RCUK to show direct alignment with government policy. (Haldane? Who’s he?)

    Philip

  10. The school syllabus does have a fair amount of the everyday material world, and I’m pleased to see it. But Horizon does not, nor has Brian Cox yet done a programme about the physics of the stuff around us in our homes etc.

    I hope you’ll forgive the shameless plug, Athene, but Nottingham’s Sixty Symbols project has focussed on the ‘physics of the everyday world’ quite a bit. We’ve done the physics of breakfast cereals, transistors, football, musical instruments (including the vuvuzela), new developments in touch screen technology, the transparency of glass, antennae…

    Sixty Symbols doesn’t have quite the ‘reach’ of the BBC (!) but the YouTube channel has picked up a reasonable number of views at this point.

    Best wishes,

    Philip