{"id":2651,"date":"2014-10-07T23:17:12","date_gmt":"2014-10-07T22:17:12","guid":{"rendered":"http:\/\/occamstypewriter.org\/scurry\/?p=2651"},"modified":"2014-10-15T14:42:14","modified_gmt":"2014-10-15T13:42:14","slug":"debating-the-role-of-metrics-in-research-assessment","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/scurry\/2014\/10\/07\/debating-the-role-of-metrics-in-research-assessment\/","title":{"rendered":"Debating the role of metrics in research assessment"},"content":{"rendered":"

I spent all of today attending the \u201cIn metrics we trust?\u201d workshop<\/a> organised jointly by HEFCE<\/a> and the Science Policy Research Unit (SPRU<\/a>) at Sussex University. This\u00a0was part of the information-gathering process of HEFCE\u2019s independent review of the role of metrics in research assessment; the review\u00a0has a particular focus on how metrics\u00a0might be used in the Research Excellence Framework (REF<\/a>) that determines block grant allocations to university departments and research institutes. I was attending because I am a member of the review steering group.<\/p>\n

The day promised to be one of vigorous debate because the consultation process that closed earlier in the summer had attracted over 150 responses \u2014 soon to be published \u2014 and these\u00a0presented a wide range of views on the dangers and potential of metrics. And so it proved to be, with three panel sessions exploring \u201cThe changing landscape for research metrics\u201d, \u201cThe darker side of metrics: gaming & unintended consequences\u201d and the question of whether there can be reasonable progress \u201cTowards responsible uses of metrics\u201d. Sandwiched between these was a bazaar in which various metrics vendors displayed their wares.<\/p>\n

I don\u2019t have time tonight to capture the range of points and insights that were offered during the course of a interesting day but was somewhat reassured by the widely shared expressions of belief that any use of metrics in research assessment \u2014 or as a probe of the propagation or impact of research into the wider world \u2014 has to be done with care<\/em>. The mantra that metrics should inform judgements and not replace them was repeated by many participants and will hopefully soon be enshrined in set of principles to be known as the Leiden Manifesto<\/a>.<\/p>\n

Almost a lone voice, Prof Dorothy Bishop presented a provocative case for supplanting the cumbersome system of peer review in the REF with a much lighter touch analysis of departmental h-indices<\/a> calculated for research-active staff \u2014 an idea that she has previously outlined on her blog<\/a>. Dorothy showed that, at least for some disciplines (including natural sciences and psychology), use of this metric generated scores that correlated well with resource allocations from the 2008 Research Assessment Exercise (the forerunner of the REF). (Update<\/em><\/strong>: as David Colquhoun points out below<\/a>, Dorothy showed today that most of the correlation is actually due to the the number of people in each\u00a0department \u2014 and she has since\u00a0detailed her proposals in a new blogpost<\/a>). The particular advantages of this approach are the cost saving \u2014 reckoned to be somewhere between \u00a360m and \u00a3100m \u2014 and the elimination of the bias that arises from panel members\u2019 affiliations. But it remains to be seen if the method is applicable across all disciplines; or if it fulfils some of the other purposes of the REF, which include examination of broader impacts and demonstrating the commitment of UK research to quality control through periodic self-examination (a feature that plays well at the level of government).<\/p>\n

I hope others might chime in with their impressions and analyses of the day. Already there is a Storify aggregation<\/a> of some of the tweets that tracked the different sessions. I include below my contribution, which was part of the session on the darker side of metrics.\u00a0It has been lightly edited to clarify and sharpen some points but remains brief and incomplete. This debate is far from over.<\/p>\n

“I come here today very much with an open mind on many aspects of metrics, though I fear that may largely be because I am still somewhat confused. So I am glad to have the opportunity to participate in today\u2019s discussions. Already, I am beginning to see some interesting things.<\/p>\n

On some topics my mind is made up. I remain sick of impact factors<\/a>, for example, because of the way that they are so commonly mis-applied in the assessment of individuals or individual pieces of research. I don\u2019t need to rehearse the arguments that I laid out in a blog post<\/a> of the same name in 2012, except to say that impact factors are a powerful illustration of how a relatively innocent innovation in quantitation can be perverted and do real damage to the research community. I don\u2019t think there is much dispute on that point (though I was surprised and disappointed to come across defenders of this metric in the discussion at the end of this session).<\/p>\n

I am worried about people being seduced by the apparent objectivity of numbers. We saw something of that last week in the excitement whipped up by the announcement of the World University Rankings<\/a> in the Times Higher Education (THE). In the preamble<\/a> to its explanation of the methodology \u00a0the THE describes the ranking process as a \u201csophisticated exercise\u201d, that is \u201ccarefully calibrated\u201d to provide \u201cthe most comprehensive and balanced comparisons\u201d. It ranks universities on a composite score drawn from estimates of a range of indicators of teaching, research volume and influence, industrial income, and international outlook.<\/p>\n

The Times Higher are good enough to be open about the methodology but when you read exactly how they assemble and weigh the various components, you read statements such as \u201cwe believe\u2026\u201d, \u201cUGs tend to\u2026\u201d, \u201cour experts suggested that\u2026\u201d or worse: \u201cthe proxy suggests that\u2026\u201d. And so you can see that, although it may be sophisticated, the measure is clearly also subjective. It is not sophisticated enough to assign error bars or confidence intervals to the scores given to universities and I think that\u2019s unhealthy. It seems as if the rankers are laying claim to a level of precision that cannot be justified.<\/p>\n

And that tendency for numerical \u2018measures\u2019 to wrap themselves in an pseudo-objective authority is a longstanding problem with metrics; in the end people adopt them without thinking hard enough about where they came from.<\/p>\n

As a result, I am worried about the word \u2018metric\u2019. It implies measurement but, although there are now an increasing number of things that we can count \u2014 thanks to the increasing computerisation and connectedness due to the internet \u2014 there is still much uncertainty (as we heard this morning from Cameron Neylon) about what those numbers are measuring or what they mean. We still struggle to define quality and impact, never mind being able to measure them. But that is OK and we should not be shy about admitting the difficulty of making judgements about quality or impact \u2014 or conceding the limitations of the things that we are counting.<\/p>\n

But I think it would be more honest if we were to abandon the word \u2018metric\u2019 and confine ourselves to the term \u2018indicator\u2019. To my mind it captures the nature of \u2018metrics\u2019 more accurately and limits the value that we tend to attribute to them (with apologies to all the bibliometricians and scientometricians in the room).<\/p>\n

As someone who is from Ireland, where we have been telling stories for thousands of years \u2014 from a time before stories were written down, never mind cited and counted \u2014 I was pleased to have heard the word \u2018story’ (or its posher cousin \u2018narrative\u2019) mentioned so many times in the session this morning. Stories matter to people and although it is now a commonplace to assert that \u2018the plural of anecdote is not data\u2019, I wonder if that is always\u00a0true.<\/p>\n

I think that in some ways the diversity of activities and qualities and impacts that are part and parcel of the academic enterprise can only be captured in stories and in narratives. We should be honest about our limited abilities to describe these attributes with quantitative indicators. More than that, we should not be shy about celebrating the wonderful stories that we can tell. I look forward to the publication of the REF2014 narratives (sorry,\u00a0stories) because I think many of us will be pleasantly surprised to find out about the different ways that research work has vaulted over the walls of academia and into the real world \u2014 where it matters.<\/p>\n

And finally, wearing the hats associated with my involvement in Science is Vital<\/a>\u00a0(SiV) and the Campaign for Science and Engineering<\/a>\u00a0(CaSE), I want to emphasise the important political dimension of the REF, which is that it provides a mechanism for the research community to demonstrate that it is accountable \u2014\u00a0<\/em>to government and to the tax-payers who fund us. I think that is important. (And I think that is it important for the researchers on the ground buy into the process and participate \u2014 it is not sufficient to leave this to provosts, vice-chancellors and research managers).<\/p>\n

With that in mind, and not forgetting the limitations of quantitative indicators, researchers shouldn\u2019t be too prissy using numbers that have some meaning \u2014 especially if they are aggregated at levels that can attenuate the noise in the system. At SiV and CaSE, the case for continued investment in UK science is based in part on the productivity and quality of our research base. In part that is estimated through numbers of publications, and citation rates. The UK has 1% of world\u2019s scientists but produces 6% of publications, and about 14% of the most highly cited papers. Do we really believe those numbers are meaningless? They are not the whole story of course. It is just as important \u2014 I am aware of the presence of sophisticated policy analysts such as Ben Martin<\/a>\u00a0and Andy Stirling<\/a>\u00a0in the room today \u2014 to be able to talk about the need maintain a research and university infrastructure so we have generative and absorptive capacity for innovation. (Not to mention the intrinsic value that research gives to human existence by satisfying our curious nature).<\/p>\n

So although there are risks, I think we should count on some indicators to inform our judgements, to test and challenge our stories (so as to mitigate our biases), and to help us tell those stories to ministers and the public. Those risks are real but I think they can be counteracted by transparency and debate. I am optimistic that the research community is up to that challenge.\u201d<\/p>\n

 <\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"

I spent all of today attending the \u201cIn metrics we trust?\u201d workshop organised jointly by HEFCE and the Science Policy Research Unit (SPRU) at Sussex University. This\u00a0was part of the information-gathering process of HEFCE\u2019s independent review of the role of … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,107],"tags":[241,312,132,313],"class_list":["post-2651","post","type-post","status-publish","format-standard","hentry","category-science","category-scientific-life","tag-hefce","tag-metrics","tag-ref","tag-research-assessment"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2651","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/comments?post=2651"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2651\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/media?parent=2651"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/categories?post=2651"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/tags?post=2651"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}