Light Touch Metrics?

In the summer metrics looked like they had been substantially laid to rest for the Higher Education sector: The Metric Tide report, written at the behest of the (probably the about-to-be-late-lamented) HEFCE took many pages to point out that metrics alone would never be a good way of assessing excellence in our universities. Whether you care to use h indices, citations, Journal Impact Factors, or pounds per anything, metrics have their limitations. Indeed, having read the report it is hard not to think that the misery of the REF can’t have been entirely bad because there was at least some sense of people making judgements based on their professional expertise, not slavishly following one number or another (with all the perverse incentives such figures can induce).

However, the government still wants to measure us, still wants to be sure their money is not ‘wasted’ and that students as consumers are not being short-changed. So assessment by metric continues to be very much in the air, whether in the context of teaching or research. I find this deeply dispiriting. If the government thinks they’re going to do things in a ‘light touch’ way as regards teaching, that is bound to mean some sort of metrics. However, if universities are going to be scored across all their courses with a single figure, what on earth is anyone going to learn from this? How will a physics course with a heavy load of laboratory work involved be turned into any sort of equivalence with a course in languages with a year abroad? If my department happens to irritate all of its students for some reason, how will that be factored into a score for the whole university? Are a lot of mediocrely OK courses across an entire university to be rated better than a university which offers a mixed bunch of high and low scores? I can see no way in which an aggregated figure can be meaningful to a student – but then it’s really designed to serve a government bean-counter deciding whether or not the tuition cap can be raised. (Which means that’s a rather important government bean-counter.)

As a Cambridge academic I have a more specific worry about how the supervision system will be factored in. This small group teaching (typically 2-3 students with a teacher) is very effective at ensuring all students get to try out their ideas and get personal feedback on whether these were right or wrong. It is very difficult for any student to sit quietly for an hour and never utter, as only too often can happen in larger groups. However, such teaching is frequently arranged through a College not the University; some of the teaching will be by professors, some by PhD students. The quality, inevitably, will be variable – and quite likely to be uncorrelated with seniority or any other obvious metric. So, how would any department ‘score’ such teaching when they have no direct control over it? And how would this then be added in to the entire University figure, given that we are a Collegiate university?

Of course one could avoid all such nasty reality by simply using the National Student Survey. As many voices have pointed out, a satisfied student is not the same thing as a well-educated student. Referring once again to the burden of laboratory work, we know it’s not loved by many students but, if they never get familiar with basic skills and equipment, how can they do more interesting stuff like research? So, a well-educated student may indeed be a somewhat unhappy student who can’t yet appreciate why they were taught some specifics. A satisfied student may be the one who gets access to all the overheads and never has to turn up to a 9am lecture at all. So, I hope the survey scores will not trump any other way of determining whether teaching is ‘good’.

Aside from government bean-counters, the other people who really care about the scores are those who devise – and utilise – league tables. Most universities, including the University of Poppleton, can find some appropriate measure by which they are in the top 10. Such league tables have proliferated. Many of them lack transparency or their figures-of-merit are not particularly helpful. By relying heavily on these we are in danger of distorting what we do and how universities present themselves.

Should universities fight against the use of numbers which do not ultimately tell us – or students or the taxpayer – about quality in any robust way? I think the answer must be yes but, as with the whole idea of successive RAE/REF assessments I fear it is a losing battle. Those individuals and institutions which have signed up to DORA (Declaration on Research Assessment) should be sure to push that its recommendations are implemented at least internally. No more h index comparisons at promotion panels, or decisions to exclude individuals from a shortlist because of the journal impact factors of their publications. In practice, I have seen the former half-heartedly operate and I guess we should be doing more. My own field, with its wildly different publishing strategies – potentially ranging from hundreds of authors in high energy physics to the lone author theorist – demonstrates clearly the importance of avoiding crude criteria. (And if you look at the list of universities signed up to DORA and don’t spot Cambridge, I am assured we are signed up through our membership of LERU which is a signatory. I do worry how many of my colleagues are aware of this. For the record, I specifically enquired about this but I am also an individual signatory.)

As the dust of the Chancellor’s Autumn Statement settles, as BIS escapes comparatively unscathed despite the bullish noises for cuts emanating from Sajiv Javid in advance of the Statement, we still have to worry as a sector what the price we will have to pay may be in terms of metrics since they look like such a convenient and cheap strategy to implement. I hope the original committee, chaired by James Wilsdon, that carefully looked at the evidence and spelled out the limitations of any metric-based measure, will stand by its conclusions.