This weekend I was persuaded by a member of my family to enter a local Parkrun. If, like me, you haven’t come across these before, I should say they offer weekly timed 5k runs at local venues. You just register, turn up and run. I am no serious runner, but once upon a time I ran a lot more than I do now, up to 50 miles a week. Then children intervened and I didn’t run at all for about 20 years, and now I’ve started again it is in a very low-key way. However, I thought I had some idea of what I was doing on a weekly basis and was satisfied with it. What this timed run showed up were my delusions, both about how far and how fast I have been regularly running. Having said that, I wasn’t that disappointed with how I performed* (sounds wonderful if I say I came second out of my category for age and gender, until I point out there were only 5 of us in that particular category), but this was achieved by pushing myself hard – and demonstrated to me just how little of that I have been doing on a regular basis. In other words, I have been kidding myself.
I suspect we can all convince ourselves, without really trying or being aware of it, that we are actually achieving substantially more than in reality we are, because that is what we want to believe. Reading some more Pathways to Impact statements later in the day – I’ve commented on these before in less than complimentary terms – I am struck by the aspirations people put down on paper, about public engagement in particular, and I worry about how often these laudable goals are actually met. (I realise I am at risk of raising the ire of Philip Moriarty again by discussing Impact once more). This isn’t meant particularly as a criticism of the authors of these documents, but it is worth occasionally confronting the gap between what one hopes to achieve, and what is actually delivered at the end of the day – or in this case the end of the grant.
As I said in the previous post, I strongly believe that quantifiable objectives are well worth laying out in these statements, something along the lines of:
• I will talk about my work at 3 secondary schools each year during the course of the grant; or
• I will present my work annually at the local Science Festival;
People rarely do so in quite such explicit ways, but a few brave people do, such as (this from a particularly well-written statement):
We anticipate one media article and one public lecture per year on our project. We expect, by the end of the project, to have created one arts-science collaboration based around its content.
To be contrasted with much vaguer, if more frequently encountered statements, along the lines of:
The applicants will also be involved in visits to local schools to give lectures on a wide variety of topics relating to…..[this grant].
A statement that would have been much improved by a qualifier such as this (from a different grant):
[We will] undertake 1 or 2 schools visits to advertise multi-disciplinary science
I am sure every PI putting such thoughts down on paper would like to feel that this is the sort of thing they really will do. But, this is where I fear the gap develops between aspiration and delivery. There are so many other things that can seem more pressing in the diary than trotting round to a local school, however much one may in principle wish to do so. Unfortunately, reading grant applications may well be one of them (note I say pressing, not more enjoyable, rewarding or desirable). My own record on the school-visiting front is not particularly impressive, having averaged one school a year recently.
However, although that record may be poor, I appear to be entering a phase where my school outreach ‘figure of merit’ will suddenly improve, not because I have signed up to a long list of school events, but because I have a couple of events lined up in the months ahead when I will be talking to quite large numbers of teachers. So, instead of reaching out to school children, of whom only a few percent will in fact take in anything that is said at all, I will be aiming to stimulate their teachers with the excitement of the physics I do. One of the issues that the Royal Society Education Committee I chair is very concerned about, as spelled out in Recommendation 5 of their recent State of the Nation Report 4, is the need for in-service training, including being exposed to current research activities and experts in their field
Recommendation 5 Science and mathematics teachers should undertake subject-specific continuing professional development (CPD) as a part of their overall CPD entitlement…..
I am not sure how talking to teachers rather than the children themselves should be ‘scored’ in the outreach stakes, but by showcasing physics research to hundreds of teachers, maybe a little something will filter down to the schoolchildren, fulfilling my own laudable if not-always-realised aspirations to be a cheerleader for science to excite the would-be scientists of tomorrow.
Or am I just kidding myself?
*If you really want to know, I did the 5km in 32:19, not totally embarrassing but hardly cause for celebration.
If I might be a little cheeky, I’m curious to know what you think are the quantifiable objectives for your upcoming talks to teachers and how your achievement of them will be assessed or measured. “Maybe a little something will filter down to the schoolchildren” sounds about as vague as the PI impact statements that you are critical of in the first part of your post… 😉
You can be as cheeky as you like, but what I wrote in my post was not what I would put in an impact statement!
You need to distinguish between the fact that I feel bad I don’t get into schools as often I think is ideal – so I’m trying to make myself feel better with the realisation that I will at least be talking to quite a lot of teachers – from any objective I personally might put down in writing in a grant application. In line with the specific example I gave in my post, I could write something along the lines of as a goal, which would be objectively quantifiable. What might be filtered down to the schoolchildren I agree is definitely a challenge to turn into a metric, which was why public engagement and outreach was something the REF Impact Pilot (I know not quite the same thing, but equally tied up with these issues) looked very hard at (see Annex G for the Physics Panel report; point 11 covers this discussion). But giving a number of people to reach, is just the same sort of measurable as the specific example of how many schools might be visited, which I was applauding. Agreed?
Most of my outreach talks are organized through national science & engineering week, for which its hard to quantify the take up by schools in advance (7 this year, 4 last year…). So should quantified outcomes of impact statements be treated any more favourably than qualitative ones? Talks tend to go down well, although putting my cynical cap on, perhaps only because they represent an hour away from normal lessons (concerns apply equally to teachers and pupils). Less skeptically, don’t science teachers who go the extra mile by inviting speakers in from uni, also represent the subset who would be expected to motivate/enthuse pupils more than the norm. Given limited time for outreach work, might it be better to target efforts on the silent majority for which external speakers aren’t often invited.
Equally, i’d not be convinced by impact statement claims asserting that a particular research project would generate “one media article” per year unless it was particularly accessible to your common-or-garden science journo. They’re a fickle bunch, plus outcomes depend almost wholly on other news on the day of the media release.
Well, yes and no. It is possible to come up with a numerical target but how meaningful is that? As I’m sure you know, it is quite possible for some people to go into a school, give a talk and completely baffle the children. So just counting the number of contacts is not necessarily very helpful – it’s the quality of that contact that matters. Which I’m sure you know (since it’s covered by point 11 in Annx G of the document you referred to ).
Don’t get me wrong, I’m not against getting people to think about impact, though I don’t think the RCs have done the best job of explaining what they want from scientists – or of countering the inevitable mixed messages. But I am against bean counting.
I’ve sat on a few committees and never known a grant to be turned down because the Impact statement was inadequate, although the applicants might have been asked to beef it up a bit in some cases. But we’d probably all agree that outreach is important, and anything that gets scientists to think about it can’t be such a bad thing. Athene (which my autocorrect keeps changing to Atheneum), do you know the Cambridge Biologists site (http://www.cambridgebiologists.org/index.htm)? It’d be good to have a Physics equivalent.
Paul – I think quantitative statements should be regarded more favourably in so far as they tend to mean the writer has actually thought about things. In the situation you describe surely you could say I will give x talks during SET week – then take-up by number of schools is not what matters but how many talks you give (assuming they are to multi-school audiences).
To address both your’s and Stephen’s comments about whether quantifiable metrics are more meaningful, I would say in my experience the opposite of bean-counting tends too often to be waffle. It needn’t be so, but it does tend to be. When I read these statements I want to gain some conviction that the writer has actually expended a few minutes thought on them and too often that isn’t the case. Of course we all know why. We hone the case for support, and then often are rushed to do all the other attachments and boxes on the form. We’ve all been there. Nevertheless, it is this sense of recycled waffle that is so dispiriting when reading a pile of applications in quick succession. And, as Stephen implies, that may well then lead to talks that are worse than useless when the PI turns up and mystifies the audience because they haven’t thought about that either. I don’t believe there is any sensible metric for the actual benefit any talk might confer!
Finally, Jim, I believe the EPSRC has been using the impact statements as part of their assessment criteria for some time, so in essence a bad one could lead to the grant not being funded. Whether that really has happened I don’t know, as I’ve not sat on a panel since their introduction. The BBSRC hasn’t done this so far but I suspect is about to change very soon; up till now, as you imply, a bad one attached to a funded grant has simply been sent back for a make-over. Other people can perhaps comment on what happens with other RC’s. Thanks for the link to the Cambridge Biologists. The Cavendish has a not dissimilar site at their Educational Outreach .
However, the point of the post was not really to attack the Pathways to Impact statements per se, but to suggest the gulf between good intentions and delivery.
I agree with a fair bit of this. My experience of these sorts of statements (of reviewing, writing and helping people with them) is that is is really difficult not to waffle. The expectations waved *at* academics are a bit loose too, which doesn’t really help. If they were clearer, and academics were given more support and training in this area, I think they’d be a lot better.
Or rather, we expect more of academics in terms of PE than we are willing to support them in. Yes, a couple of talks is realistic. But it’s not good enough. So, much as I agree that people shouldn’t kid themselves, I’d also say that I don’t think a few schools talks and festival presentation *should* cut it. That’s not to say that you personally should spend more of your time on PE, but the overall PE output of your research should be a lot greater. It should be meeting the sorts of lofty aims we hear a lot about.
Personally, I think that if we are serious about PE then all large research projects should have a professional public engagement officer imbedded as part of the team, just as they have professional administrative support. Their job wouldn’t be to do PE separately, but to help and inspire the academics do the PE work themselves – simple things like setting up a blog and helping run it. Some do this already, though many just pile it on as yet another job for academic or administrator who (a) doesn’t have time and (b) is struggling a bit with the skills. I know some sites do this a bit already, I suspect many more if they’d get the funding. If you want this stuff done well, you need to pay for the staff.
(I’m side-stepping the point that no one actually cares about it though – as comment above says, it’s rare research gets turned down for bad PE statement).
I just wrote one of these statements for a grant – which is probably pretty waffly – one of the problems I find with outreach is, like you state, not that I don’t want to do it but really HOW to do it in any realistic way. There is not much guidance out there on how to even write a pathways to impact other than the Venn diagram thing they give you – where they also tell you basically not to over egg the pudding; much less on what would really be effective.
For instance, who do I get in touch with, how do you get in touch with people to outreach to? there are departments in some Universities for this – but not all and not all that are effective. so people are stuck with waffling, in some instances – and as for Alice’s suggestion to employ people – its a nice thought but is it one that they would be willing to fund you for this? Given that for some people its between cutting a PDRA position to get research done or paying someone for PE – most people would probably go with the PDRA; I would – mostly because you have to have the research to do the PE – the chicken definitely comes before the egg.
Maybe Pathways to Impact is just going through some teething problems – and all of this will get sorted out in the future.
Quantifiable outreach, while laudable and necessary from a report-writing point of view, may not be the same as successful outreach. I sometimes think that there is a bit of the ‘faith of the mustard seed’ in outreach efforts. For example, how do I know who Athene Donald is? I happened, a few years ago, to have the radio on while she was being interviewed on Desert Island Discs. I heard a woman speaking about being enchanted by physics while at school and I was hooked. I was similarly enchanted at school. How powerful to hear a female voice expressing these thoughts. I am now a teacher and find myself looking for ways to inspire my students not only with the beauty of physics itself, but also with stories of those who have made a life pursing a deeper understanding of it. Catching the last 15 minutes of a radio show really changed how I think about possibilities for women in physics. One of the things I have come to learn about being a teacher is that I can’t really know the impacts I have. Certainly there are quantifiable measures, but I believe those are not necessarily the same as the impacts that make the most meaningful differences. Perhaps when you speak to the large number of teachers later this year, that ‘little something that filters down to the schoolchildren’ will be as much about the example of being a physicist as it will be about the physics itself. While that may be nearly impossible to quantify, those are precious seeds to sow.
Perhaps it is a good idea to go to the introduction of your story and look to the roots of the impact you made on the world of running. You wanted to take part in a run, no matter how you performed – so you had a go. I suspect that most people say all sorts of things in impact statements with no real intentions of taking part in the outreach activity, or web blog (…!). If people “had a go” though they would find it a fulfilling excercise, even if they felt they didn’t perform too well. I’ve done a number of schools talks/public lectures/radio shows, some went well, others badly wrong. What I can say is though they helped me to think about my science (sounds like a cliche, but it’s true) and how it might come across to the public. “Is what I do worthwhile?” is not a bad place to start with an impact statement. Show your statement to someone outside of the discipline, a non-scientist, and see what they think. I think setting yourself targets is a good idea as well. I certainly have to do that with exercise, so why not with a set of goals in impact? Some people are not predisposed to take part in some outreach activities i.e. stand at the front of the class and say how great science is, but may be able to contribute in other ways, less public than this.
The overarching message that seems to be coming through the different comments is: do anything; you’ll never be able to judge what effect you may have had, but that’s still better than doing nothing. Writing something down will still require some thought as to what is appropriate for each individual and each project – and I come back to the fact that the very act of thinking about these statements is important whatever the outcomes.
Thanks Melissa for your specific encouragement. You mentioned Desert Island Discs, and I always felt that had to be the most effective outreach I have ever done. I was astonished by how many emails I got from complete strangers saying how what I said had touched them in some way or another. Equally – not only was that media appearance completely unpredictable, and so couldn’t have been written down in advance – I only got to say about 2 sentences about my actual science as opposed to being a scientist. Perhaps that is something it’s too easy to forget: people (probably especially school children) have so little idea what a life in science is all about that just discussing the realities of what we do and how exciting it is, is as important as any technical details.
I find one of the difficulties with impact statements, and particularly with attempting to apply quantitative metrics to work that hasn’t yet been done, is the need to be project-specific. Wouldn’t it make more sense on RC application forms, at least when it comes to PE, to ask more general questions such as ‘what is your track record in PE’, and ‘what sort of things do you do to engage the public’? It is highly unlikely that I would ever go into a school to talk about work linked to an individual project, but I do occasionally talk to schoolchildren in a much broader and generic way about medical research. And although there is a ‘schools’ focus to the thread here, there are of course many other audiences who may be interested in your work. Certainly for me, in eye research and blindness, the largest and most interested public group is not schoolchildren but the elderly.
The difficulty of predicting impact, as requested on RC grant applications, is brought into focus by a recent publication by the Rand Corporation entitled Project Retrosight, an unbiased analysis of the long-term impact of a clutch of research grants funded some 20 years ago. This is well worth reading regardless of what one thinks of the ‘impact agenda’, not least because it reveals the complexity of impact analysis even when performed retrospectively. What I kept asking myself as I read the document was, ‘how much of this could have been forecast by the applicants at the time they submitted their proposals’? Whatever the answer, the content did provide me with several useful ideas for the impact sections of my own recent RC grant application, in which I didn’t mention schools at all (though perhaps I could have) and chose not to try and quantitatively predict any prospective measure of impact.
@steve: re predicting impact, you might remember this post, and the piece of Tim Biscoe’s in Physiology News that it links to. The work of Comroe & Dripps that Tim talked about did the same kind of thing that Rand has done, though in an earlier era. They were also quite clear on how little of the impact of biomedical research could have been predicted.
Yes I do remember this, though I somehow missed your September post. Digressing slightly I briefly overlapped with Tim Biscoe when I first came to Physiology. At that time there was some resistance to my appointment from some who felt that molecular biology had no place in a Physiology Department. This led to a challenge to Tim to define Physiology, to which his measured response was ‘that which goes on in a Physiology department’.
There is indeed much good evidence that impact cannot be forecast, so the question persists as to whether it is of any value to try. I think not, but when obliged to do so by those who fund our work there is little choice.
Heh. I like Tim B’s response – I can just hear him saying it..!
I guess a more expansive way to phrase the same sentiment would be “anything that will advance physiology research and teaching”.
We have had analogous arguments here, especially (!) with respect to “what constitutes pharmacology”. The pharmacology gang seem to be particular prone to this kind of thing – I have come across a fair few over the years by whose definition even someone like David Colquhoun would not have “counted” as a pharmacologist.
I suspect all disciplines are the same. My previous head of department Sir Sam Edwards, when challenged about some of the odder food physics we were doing in the department, used to say ‘physics is what physicists do’.