Reciprocal Space

Berlin Debate – Who owns science?

Posted on November 8, 2019 by Stephen

Yesterday I had the privilege of participating in the 14^th Berlin Debate on Science and Science Policy, which is hosted by the Robert Bosch Foundation. This year the invited participants discussed “Who owns science? Reshaping the Scientific Value Chain in the 21st Century“. There was a lot to unpack in that topic but for my formal contribution as chair of DORA (a six-minute opening statement), I took the opportunity to explore the links between open science, research evaluation and inclusion. My statement is reproduced below:

I have been asked to look at one of the key challenges on the road to Open Access and Open Science: How to secure and measure quality if we stop relying on the Impact Factor (and the peer review-systems) of prestigious, but paywalled journals.

The first part of that question, how do we secure quality, presupposes that the present system of scholarly publishing does a good job of quality control. Now I am not a fan of talk of a “reproducibility crisis” or of proclamations that “science is broken”; but clearly, despite the fact that many authors, reviewers, editors and journals are doing a good job, there are problems.

Our systems of research evaluation have become over-metricised. In particular, as we all know, we have an over-developed reliance on journal prestige – most often signified by the impact factor – as a measure of achievement.

This means that the dual functions of publication, first to share new discoveries and insights, and second to establish a reputation on which to build a career, have become unbalanced. The chase for a metricised reputation panders to selfish instincts and stokes hyper-competition by focusing on objects (papers and grants), rather than on people.

It has led to numerous perverse effects that undermine science:

problems with the reliability of the scientific record;
increased retractions, which undermine public trust;
delayed publication which, if the public was aware of it, would further undermine trust;
we have a very patchy record on data and reagent sharing;
and a publication system that is still largely closed, not just to the public but to many academics.

Our obsession in research evaluation with metrics such as impact factors is an unhealthy breach of science’s social contract and increasingly puts scientists in conflict with the values that many claim underpin science.

Those values are rooted in the moral purpose of science, which is to make the world a better place – for everyone. They have emerged over centuries of continuous discourse. I don’t have time to excavate the history in detail so let me be content with a single footnote from Robert Merton’s famous 1942 paper on science and democracy, where he highlights remarks made by the 18th century French entomologist and thinker René Antoine Ferchault de Réaumur. Writing about the moral compulsion to publish one’s research, and while acknowledging national obligations, Réaumur ultimately concluded that “those who work to advance science and the arts should consider themselves citizens of the world.”

To me this is a 300-year old vision of open science. And it is my conviction that open science offers a route out of many of our present difficulties. It provides opportunities to reconnect with our most fundamental values, not just as scientists, but as human beings.

To be sure, there are some practical difficulties to be overcome in moving toward open science practices (inequities of cost, for example, that I hope we might explore in the next session), but there are no fundamental impediments and no serious threats to the quality of our research work. In fact, in very many ways, an open approach will improve our work because it demands that we expand our view of the qualities of research that matter and that should be evaluated.

The rise of preprints, for example, makes science faster; and because they present results independently of any journal brand, they provide a healthy focus on the quality of the content. Their openness (as with open access papers) means that they are available for the widest possible scrutiny – an incentive to authors to get things right – and for the widest possible re-use. Preprints also appear to foster commentary in ways that peer reviewed publications have struggled to do and this is helping to pave the way for experiments in open peer review (a practice that, if adopted universally, would kill off predatory open access journals).

Just as importantly, the rise of open science raises the question of why academic papers should be available to all. It is not just, as many governments reasonably contend, that the public has the right to see what they have paid for, and to demand maximal returns on its investments, but also because there are many stakeholders with a genuine interest: patient advocacy groups, environmental campaigners, citizen scientists – indeed any citizen concerned about the direction of travel of the modern world.

The dialogue engendered by open science raises the question of who should shape science. This is not just about engaging with the public, enabling them to have a say in setting the research agenda*. It is also, crucially, about taking a closer look at the fact that in no country that I can think of is the academy representative of the population at large. Women and ethnic minorities, among other groups, are chronically under-represented. This is in part a result of our present system of evaluation: metrics and rankings too easily sustain the status quo. How can an academy that is not truly global claim to be able to identify the world’s most pressing problems?

Open science compels us to re-examine what it is that we – the diverse citizens of the world – want from science, and to figure out how we are going to evaluate that. This is a project that DORA is keen to advance, through advocacy and speechifying, yes, but also by developing and promoting practical solutions.

To me open science represents hope: a chance to re-negotiate science’s social contract in ways that are more inclusive, more open to public dialogue on the proper balance between academic freedom and responsibility, and more rooted in the moral purpose that so many of us are seeking to give meaning to our lives.

Posted in Science | Comments Off

First impressions: the DORA-HHMI meeting on research assessment reform

Posted on October 27, 2019 by Stephen

My feet have hardly touched the ground since I got back from the DORA-HHMI meeting on “Driving Institutional Change for Research Assessment Reform” in Washington DC last week, but I wanted to log a few first impressions. I can’t hope to do it justice but please be reassured that the videos of the plenary session will soon be available, and there will no doubt soon be wiser and more considered ruminations on the conference.

Economist, Paula Stephan

From Monday evening until Wednesday lunchtime (21-23 Oct), we gathered in the plush surroundings of the Howard Hughes Medical Institute headquarters to work through the present challenges in research assessment. I was there in my a capacity as chair of the San Francisco Declaration on Research Assessment (DORA). It was a good size of meeting – about 60 participants; small enough so that you could clock the faces of most of the people there over the three days, but large enough to have a good mix of stakeholders, including vice-presidents for research, faculty deans, researchers, librarians, representatives of learned societies and funders, and the odd economist and behavioural designer.

At meetings on the topic of resesarch assessment, there is a tendency for the discussions to be orbit around the same old problem – the pernicious effects of journal impact factors and the chase for position in university rankings. But I am pleased to report that there was a conscious effort here to direct attention to solutions. This effort was nevertheless rooted in a clearly articulated determination to recall the values that are the foundation of scholarly research, which date back at least to the work of Robert Merton, but are also subject to the questions raised by the open science movement and the increased attention being paid to issues of equality, diversity and inclusion in the academy. These values recall not only our duties within the academy as scholars who subscribe to communitarian principles, but also our responsibilities as mentors, leaders, and managers, and as professionals who started out with a keen sense of wanting to make the world a better place.

The problem with values, as became apparent in our deliberations, is that they can too easily disappear amid the everyday pressures to get the job done. An evaluation system preoccupied impact factors, citations, grant income and league tables soon distracts people and universities from their higher aspirations. How best to keep sight of our larger purpose?

Me, trying to keep up with Frank Miedema…

There was a clear-sighted recognition at the meeting that reform of research evaluation has to be thought through in very practical terms. Values on their own are not enough. For one thing, we need to ensure that university leaders are aware of the problem associated with over-reliance on simple metrics. There is too much of a “Yes, but…” culture that pays lip-service to the problems without tackling the root causes.

As we heard, tackling the root causes requires institutions to have honest conversations about their values – perhaps best conducted at more granular – departmental? – level within the organisation where everyone can be involved. This is key to empowering researchers, from PhD student to professor, to take ownership of the problem of resesarch assessment.

But more than that, we need workable alternatives to the traditional metrics of ‘excellence’, assessment mechanisms that take a broader, deeper view of the aspects of scholarship that matter, that embrace not only the diversity of important outcomes from research (the particular quality of the work, as well as associated outputs such as data, software, trained researchers, etc.), but all the other important activities that academics undertake, such as teaching and mentoring, departmental and disciplinary service, and public and policy engagement. Here there were a number of interesting examples of good practice, including the use of biosketches to capture a more narrative-based summary of individual accomplishments, and structure interview techniques that try to probe beyond the desire to generate papers, for example by testing candidates’ commitment to diversity or their abilities as team leaders. Beyond that we had fascinating and provocative discussions on the economics of science, and the importance of deliberate and systemic approaches to organisational culture change.

Dinner and discussion at the DORA-HHMI meeting

The meeting was intense and invigorating, and many of the discussions were tough. At times they were challenging and uncomfortable – an important path to learning (though everyone still seemed to get along in the evening socials in the bar!). Personally, I am grateful for the insights from vice-presidents of research that I spoke to about the perverse and damaging, but inescapable pressures they come under from university rankings.

The realities of research assessment are harsh and it is clear that, despite lots of interesting and innovative experimentation in this area, we are still figuring out what works best. One of the most important lessons of the meeting is that that’s OK. For sure, we will have to evaluate what reforms work best, but it is good that people are already experimenting and innovating. It will take time to find out how best to escape the perverse effects of over-metricisation, but it is already an important signal to the research community and to society at large that we are trying to do better.

I hope that, like me, most of the meeting participants have returned home not just energised by interactions with a people who committed to change, but equipped with an expanding repertoire of ideas about how to make change happen where it matters: on the ground.

If I have seemed lax in crediting ideas to individuals, it is because much of the meeting was conducted under Chatham House rules which prevent such attribution. But I am more than happy to give credit to those whose written work in this area resonated strongly in the meeting. I list a few of them below as suggested further reading on values and practical approaches to research(er) assessment. I will be happy to add further suggestions – leave a comment or ping me an email.

Needhi Bhalla, (2019) Strategies to improve equity in faculty hiring
Sandy Schmid, (2017) Five years post-DORA: promoting best practices for research assessment
Meeting participant commentary, 10 blog posts summarising key issues in research assessment reform
Moore et al., (2017) Excellence R Us: university research and the fetishisation of excellence

Posted in Science | Comments Off

I told myself…

Posted on August 21, 2019 by Stephen

I am on holiday – in Mauritius, a tiny tropical island in the Indian Ocean. And when I go on holiday, I make ridiculous plans.

I told myself I’d start running again. I’d get up early every morning, slip out of bed without disturbing my wife and cross the road to the beach. It might be hard at first but I could jog along the compacted sand at the shoreline, the sun rising at my back. It would be good for me.

I told myself I’d write every day. I’d get up early, slip out of bed without disturbing my wife, make a pot of coffee and sit at the table on the patio overlooking the tropical garden. It would be time just for me; a chance to get back into the groove now that I have a week free from the agitation of work.

I told myself I’d read four books. I packed Philip Ball’s Beyond Weird, William Goldman’s Adventures in the Screen Trade, Bill Browder’s Red Notice, and Manhattan Transfer by John Dos Passos. (Is it that my interests are broad or that I lack focus?)

I told myself the holiday would be a chance to reflect, to dig deeper into some of my preoccupations at work. I would have the time to really think about how to weave together the elitism of a research-intensive university with the equality of opportunity that the integrity and values of the idea of the university would appear to demand. And how to achieve that in the midst of a society that claims to appreciate variety but has never really been comfortable with difference.

I told myself I would have time to muse on what more I can do to advance the aims of DORA, the Declaration on Research Assessment (which I chair). The argument for reforming research evaluation is technical and slippery. How to inspire and facilitate changes that will truly bring out the best – defined as broadly and meaningfully as we can – in scientists and scholars?

I told myself I’d avoid emails, and Brexit, and social media.

Five days into our holiday, I have done almost none of this. No running, no writing beyond this scribbled post, and precious little in the way of deep thought. I have finished Philip Ball’s book on quantum mechanics and started Goldman’s gossipy account of life as a Hollywood screenwriter. I have lacked the discipline to keep away from emails altogether – though keeping in touch while giving myself permission to ignore work has helped my equilibrium. I have tried my best to wrestle free of twitter and Brexit but they do clamour so.

But I have succeeded in swimming in the turquoise sea that surrounds the island. I have sat on the beach listening to the rumble of the white lines of surf breaking over the distant reef, while the water swishes and laps at my feet. I have felt the bloom of the sun’s heat on my face as the blazing orb emerges from cloud cover, and seen its light turn the sea and sand to silver and gold. I have chatted and reminisced and laughed with my wife and children over cocktails at a beachside bar. I have been surprised to find Jupiter in the southern night sky almost directly overhead.

So much for plans, but that will do nicely for now.

Posted in Blogging, Scientific Life | 2 Comments

The unsustainable goal of university ranking

Posted on May 20, 2019 by Stephen

Ranking organisations are seeking to diversify the measures use to evaluate universities. But without addressing the fundamental flaws in their methods, they will crush rather than embrace the rich complexity of our institutions of higher learning

When the Times Higher Education (THE) released their University Impact Rankings back in April, the idea of scoring institutions on the basis of their contributions to the UN sustainable development goals was hailed by THE’s ranker-in-chief, Phil Baty, as ground-breaking. The new rankings are certainly innovative, but for many academics it seemed like business as usual. Here was yet another way to slice and dice the data on university performance from a global industry that, although non-existent 20 years ago, has become a permanent feature of the higher education landscape. New rankings come so thick and fast these days – Wikipedia lists no fewer than 24 different league tables – that the temptation is to shrug and pass on to the next thing. But that would be a mistake.

On the face of it, THE are to be commended for diversifying the scrutiny of university performance beyond the debatable measures of reputation, research prowess or teaching excellence that are most commonly used to compile rankings. This is a sensible move. But fundamental problems remain which afflict all university league tables and are too often overlooked by student and universities – the major consumers of rankings. Students might be forgiven for not appreciating the uncertainties and incompleteness of the analyses that pull university rankings dangerously close to the realm of fake news, but universities should know better.

I suspect that most do know better but feel compelled nonetheless by market pressures to jostle for position in rankings that originated in the news media and are still granted widespread coverage. I don’t doubt the rankers’ belief in the valuable service they see themselves providing to the sector; and I’m sure they can justify to themselves the hard-nosed, real-world pragmatism brought to the task of scoring university performance, which may even be shared by some university leaders. But we can’t afford to side-step the fractured logic at the heart of league tables – the simplistic quantification and aggregation of incomparable qualities – which tears at the intellectual integrity that is surely still central to the idea of the university. We need to find better ways to set the standards for the multiple and various dimensions of university life so that, as they evolve in a constantly changing world, they can continue to enrich the societies and communities in which they are rooted.

And now perhaps, it is the turn of the rankers to shrug at the ramblings of yet another disgruntled academic. It’s all too easy to criticise and carp, they might say, but what about solutions? Students and governments rightly want to know about standards of performance, so who is going to hold universities to account?

Well here’s an idea: let’s try do it together. Cynics, please look away now. Let’s take at face value the higher aspirations for the health and impact of universities found in the mission statements of rankers and our institutions of higher learning, and proceed in the hope that they might help us jointly to unpick the tangle of political, commercial, and institutional interests that sustains the rankings industry in its present form.

Even if there is the will to do that – and I concede it remains an open question – the technical challenges are formidable. But before we can devise solutions, we need to understand the problem. We can start to get to grips with that by looking at how the THE put together its University Impact Rankings. To their credit, the THE provide a detailed description of how how they go about scoring universities. Let’s look at the big picture first. Here’s the summary of how the overall rankings are compiled:

“We use carefully calibrated indicators to provide comprehensive and balanced comparisons across three broad areas: research, outreach, and stewardship.

“There are 17 UN SDGs and we are evaluating university performance on 11 of them in our first edition of the ranking (click on a category below to view its specific methodology):

SDG 3 – Good health and well-being

SDG 4 – Quality education

SDG 5 – Gender equality

SDG 8 – Decent work and economic growth

SDG 9 – Industry, innovation, and infrastructure

SDG 10 – Reduced inequalities

SDG 11 – Sustainable cities and communities

SDG 12 – Responsible consumption and production

SDG 13 – Climate action

SDG 16 – Peace, justice and strong institutions

SDG 17 – Partnerships for the goals

Universities can submit data on as many of these SDGs as they are able. Each SDG has a series of metrics that are used to evaluate the performance of the university in that SDG.

Any university that provides data on SDG 17 and at least three other SDGs is included in the overall ranking.

As well as the overall ranking, we also publish the results of each individual SDG in 11 separate tables. This enables us to reward any university that has participated with a ranking position, even if they are not eligible to be in the overall table.

A university’s final score in the overall table is calculated by combining its score in SDG 17 with its top three scores out of the remaining 10 SDGs. SDG 17 accounts for 22% of the overall score, while the other SDGs each carry a weighting of 26%. This means that different universities are scored based on a different set of SDGs, depending on their focus.”

Setting aside the question of what exactly is meant by “carefully calibrated”, the first major problem with this approach is the arbitrariness of the selection of SDGs. Only 11 of the 17 goals are included. The missing SDGs are:

- - SDG 1 – No poverty
  - SDG 2 – Zero hunger
  - SDG 6 – Clean water and sanitation
  - SDG 7 – Affordable and clean energy
  - SDG 14 – Life below water
  - SDG 15 – Life on land

The exclusion of these goals was presumably part of the careful deliberations that went in to the construction of the tables but they are hard to figure out. If your university is doing work on economics or social policy that might tackle poverty, that doesn’t count. The same applies if your institution is active in areas of food or agricultural technology, engineering solutions to the supply of clean water or clean energy, or if it researches environmental issues or biodiversity. Research and education in all of these domains have the potential for world-changing impact but this is effectively given zero weight by the THE. Perhaps there are technical reasons for the omission – a lack of sufficiently relevant indicators or data – but these are not given in the methodology. The aggregate estimation of impact by the THE is therefore incomplete.

The second major problem, which is more serious and more striking, is the non-comparability of the overall scores, since these are based on quantification of different activities at different universities. The impact score of a university is its SDG 17 score added to their highest scores for three other SDGs. This is a pragmatic rather than a scientific choice. And given that what we are talking about here are universities, supposedly society’s stoutest bastions of scholarly critique, it’s an astonishing one. The methodology is even more arbitrary than ‘standard’ league tables that at least score universities on a common set of categories (an approach that still fails to address the issue of assessing overall performance from non-comparable attributes – see below). It’s the equivalent of trying to figure out who is the best at sports by compiling a single league table that ranks footballers, tennis players and racing drivers. There is no intelligent way to do it. In the domain of sport, what’s the harm? The rankings would generate stories that people would have fun debating. But do we not take our universities more seriously? It is hard to escape the suspicion that news values are behind the desire to reduce university performance to a single number. Rankers can and should do better.

Who’s the best? (Original photos via Wikipedia – click image to see links)

Third, even within the scoring mechanism for each of the different SDGs there are issues of arbitrariness and incompleteness. For example, below in outline is how SDG 3 – good health and well-being – is scored. The ranking focuses on a number of disparate but related elements: “universities’ research on the key conditions and diseases that have a disproportionate impact on health outcomes across the world, their support for healthcare professions, and the health of students and staff.” The total score for SDG 3 is totted up from the following components, which track both inputs and outputs:

- Research on health and well-being (27%)
  - Proportion of research papers that are viewed or downloaded (10%)
  - Proportion of research papers that are cited in clinical guidance (10%)
  - Number of publications (7%)
- Proportion of health graduates (34.6%)
  - “proportion of graduates who receive a degree associated with a health-related profession out of the institution’s total number of graduates.”
- Collaborations and health services (38.4%)
  - Collaborations with local or global health institutions to improve health and wellbeing outcomes (8.6%)
  - Outreach programmes in the local community to improve health and wellbeing (8.6%)
  - Free sexual and reproductive health services for students (8.6%)
  - Free mental health support for students and staff (8.6%)
  - Community access to university sports facilities (4%)

As with all such measures, the assigned weightings are arbitrary (and in this case, inexplicably precise). The selected components seem like reasonable targets but what exactly is being measured here? For example, how does one score a university’s provision of “free mental health support for students and staff”? Is there any measure of the standard of service – or the mental health improvement of the recipients? The THE asks for supporting evidence; this is “evaluated against a set of criteria and cross-validated where there is uncertainty.” But the criteria are not given, and although it is good to see acknowledgement of the uncertainties in the information being used to compile scores, there is no estimation or reporting of these uncertainties in the rankings – a long-standing problem that has been raised before but which no ranking organisation has properly addressed.

The scoring of SDG 12 – responsible consumption and production – takes a different approach. Here the focus is on “efficient use of resources and minimising waste”. The total score comprises the following components:

- Research on responsible consumption and production (27%)
- Operational measures (26.7%)
  - Policies on ethical sourcing of goods (4.9%)
  - Policies on the appropriate disposal of hazardous waste (4.9%)
  - Policies on minimising waste sent to landfill/maximising recycling (4.9%)
  - Policies on minimising the use of plastics (4.9%)
  - Policies on minimising the use of disposable items (4.9%)
  - Evidence that these policies also apply to outsourced services (1.1%)
  - Evidence that these policies also apply to outsourced suppliers (1.1%)
- Proportion of recycled waste (27%)
  - Proportion of waste that is recycled (13.5%)
  - Proportion of waste that is not sent to landfill (13.5%)
- Publication of a sustainability report (19.3%)

Again, while each of the components seems a sensible choice, the score is made up of disparate elements assigned arbitrary weightings. Why should publishing a sustainability report earn almost as much credit as operational measures? There’s no good answer to that – it could be debated endlessly.

As a single estimate of activity in this one area, it may be reasonable to suggest that this method for evaluating progress towards SDG 12 is good enough to win a consensus of sorts among people keen to get on with the job of ensuring that the university uses resources well. But we still have to come back to the fundamental problem with the THE’s overall impact ranking that it depends on an arbitrarily weighted sum of the scores for disparate activities, each of which is the arbitrarily weighted sum of the scores for disparate activities. It is in this unreasonable aggregation that the system falls to illogical pieces.

And remember – these are just the impact rankings. While they might represent a well-intentioned attempt to diversify the benchmarking of university performance and to pull attention away from the narrower focus of standard rankings, I wouldn’t want to see them aggregated into the single score use in the THE’s Global university rankings (and am not aware of any plans to do so). Of course, this means that when the 2020 global rankings are published later this year, much of the attention recently devoted to trying to recognise impact will be lost. That raises the question of how serious the THE is about giving people a truly holistic view of how well the universities of the world are fulfilling their different missions.

There is a better way. That is to include all reasonable estimates of valued university activities – such as their academic and societal impact, and the quality of internal processes such as staff management, resource management, and the various dimensions of student experience, but without aggregating the data. Rankers need to embrace the full complexity and diversity of what universities do, while at the same time being more open about the uncertainties in the measurements and the incompleteness of their analyses.

This disaggregated approach has already been adopted by the Leiden Ranking generated by that university’s Centre for Science and Technology Studies (CWTS). As such, it is an embodiment of the CWTS principles for the responsible design, interpretation and use of university rankings. The Leiden Ranking may more narrowly focused in scope than the suite of rankings generated by the THE, but like them it has started to incorporate elements that go beyond purely academic impact, such as gender balance and institutional commitment to open access. Even with disaggregation, one has to think carefully about institutional and disciplinary contexts to make full use of these data, as Cassidy Sugimoto and Vincent Lariviere have shown in a recent analysis of the gender data on academic authorship. Such care and insight fades from view when university ‘performance’ is boiled down to a single number, and lost from sight altogether amid the headlines of this or that university moving up or down the resultant tables.

There are risks in expanding the number of measure of university performance (even if one adheres to the principle of disaggregation). The burden of measurement – which seems to have an inexorable tendency to grow – may soon begin to outweigh the benefits of evaluation. There is something dehumanising in seeking to put a number on every particle of human activity, however worthy the aim; and finding the appropriate balance between quantitative and qualitative modes of evaluation is task that demands constant vigilance and negotiation. But the risks should be mitigated by involving the measured in co-designing the processes of evaluation. The trick is to find effective ways for different stakeholders to do so in good faith.

I believe Phil Baty is sincere when he says “Universities make the world a better place in so many different ways. @timeshighered is delighted to champion that work…” That is a vision shared by many who lead and work in universities – and one that surely has an enduring appeal to students. But the credibility of that vision depends critically on our ability to develop a collaborative approach to delivering it. If you read the rest of Baty’s tweet it says “…with the new University Impact Rankings, out in April #THEglobalpact #SDGs”. The key question for Baty – and all university rankers – is how willing they are to take the risks of engagement with the sector, and the serious critique of their methodologies. Consensus may not be possible – academia is perhaps too fractious and ranking organisations are clearly constrained by their commercial interests – but think of the impact we might have if we could find a sustainable way to work towards a common goal.

Posted in Research Assessment, Science & Politics, Scientific Life | 1 Comment

How to value what cannot be measured?

Posted on April 10, 2019 by Stephen

This post is a transcript of my opening remarks at the a Great Debate held earlier today at the European Geosciences Union 2019 meeting in Vienna. The debate asked us to consider the question: What value should we place on contributions that cannot be easily measured?

Update (13/04/2019): A video of the whole debate is now available online. My opening remarks start at 20:43 but if you have time I would recommend listening to the whole session.

As scientists, measurement is what we do. It is how we have built our disciplines and won the admiration and respect of our peers and the public for the many wondrous ways in which we have illuminated the world.

It is only natural therefore that we would seek to turn our rulers and compasses on ourselves.

But while numbers have done so much to help us understand the natural world, they are far more difficult to apply to the more complex world of human affairs, science included.

So we are here today to debate this question because of the stresses and strains that an over-enthusiastic and ill-considered application of numbers – or metrics – in research evaluation has brought to the academy.

Now… when trying to figure my way out of a problematic situation, I like to think about death. And I would encourage you all do do the same.

This quotation is number 1 of the top five regrets of people who have reached the end of their lives (according to a book by former nurse Bronnie Ware). I suspect it resonates, perhaps a little uncomfortably, with many of us in this room.

The difficulty we face every day is how to be true to ourselves. How do we cling to our ideals and highest aspirations in a world that seems daily to distract us from them? This problem is perhaps particularly acute for academics because we need to forge a reputation for ourselves if we are to succeed in our careers.

And the problem with that, according to Thomas Paine, is that a reputation is what others think of us – how they evaluate us. It does not seem to be enough to be true to oneself. We have to convince others or our scientific worth.

And the problem with that, is that our system of research evaluation as a whole seems to have made us prisoners of numbers.

We have been captured by citation counts, JIFs, and h-indices. And while these indicators are not entirely devoid of information that might be of some utility in research evaluation, they have taken over to an extent that is dangerous to the health of scientists and to the health of science.

In brief we can list the problems with metrics:

reduced productivity (as people chase JIFs in rounds of submission and resubmission)
displacement of attention from other important academic activities
focus on the individual that undervalues the role of teams
positive bias in the literature – the reproducibility (or reliability) crisis
focus on academic rather than real world impacts
hyper-competition that preserves the status quo (what chance diversity?)

As Rutger Bregman writes: “Governing by numbers is the last resort of a country that no longer knows what it wants, a country with no vision of utopia.” We might say the same of academia.

Now it’s easy to be critical of the mis-use of metrics… it’s much harder to come up with solutions.

Bregman’s prescription for change is radical. He exhorts us to be “unrealistic, unreasonable, and impossible”.

And I completely agree. But I also think that to change the world we also have to be realistic, reasonable and think about what’s possible.

(What can I tell you? I’m a mess of contradictions.)

So it’s easy to be a critic. It’s harder to think through the problem of evaluation creatively and constructively. But that’s precisely what we need to do if we are to take a properly holistic approach.

There are already plenty of bright minds thinking about this.

The UMC Utrecht is one of a number of institutions that has reformed its hiring and promotion procedures to create practical space for qualitative elements in assessments. Applicants have to write a short structured essay addressing their contributions on 5 fronts:

research
teaching and mentorship
dept citizenship
clinical practice
entrepreneurship and public engagement

This gives reviewers information in a structured, consistent and concise form that embraces quantitative and qualitative aspects of academic achievement. It is an honest and practical embrace of the principle the ‘science quality is hard to define and harder to measure’.

We are never going to escape the problem of research assessment – a world in which resources are finite demands it of us (to say nothing of public accountability). But while we will always be tempted by metrics – to the simplifying power that they seem to offer – we have to resist this. As Gawande writes: “We yearn for frictionless, technological solutions. But people talking to people is still how the world’s standards change.”

So I am glad that we are talking about this today. It is only through discussion – in good faith– of what really matters to us, through negotiation and and through the recognition of the importance of good judgement that we will come to a better definition of what success in science looks like.

I look forward to continuing that discussion in the Q&A.

Posted in Academic publishing, Open Access, Science, Science & Politics | 2 Comments

Thinking globally about research evaluation – LIS-Bibliometrics talk

Posted on January 31, 2019 by Stephen

Last Tuesday I attended the 2019 LIS-Bibliometrics meeting which focused on open metrics and measuring openness. I was part of a panel that discussed the topic “Thinking globally about research evaluation: common challenges, common solutions”. Chaired by Lizzie Gadd from Loughborough University, the panel also included Ian Rowlands (King’s College) and Kate Williams (Harvard University/Cambridge University).

To break down the topic, each of us was asked four questions. Below are the answers to these questions that I prepared for my opening remarks. I’ve left in the bold-face type the points that I wanted to emphasise.

(Ideally I would attempt here to synthesise the discussion but I’m afraid I haven’t had the time to do so. The central problem remains of constructing processes that can reasonably and fairly incorporate quantitative and qualitative information in research evaluation.)

In your view, what are the main research evaluation challenges we face?

Biggest challenge: How do we temper the tendency to competition (and indeed to hyper-competition) with being kind to people?
Next biggest challenge: The pervasive and insidious effects of metrics or indicators. Indicators provide someinteresting information, but the utility of that information is hard to evaluate. We need to be mindful of keeping the numbers in their place, remembering that complex human activity cannot be boiled down to numbers. And that is genuinely hard to do.
But I believe it can be done by rooting our procedures for assessment in our values. We need to keep asking ourselves what are for us the most valuable attributes of research and researchers? Along with many others I would like to see those values broadened out.
Yes, we want exciting and innovative research. No question.
But we also want to incentivise the sharing & re-use of that research through OA and through sharing data and code.
And we want to reward not simply novelty but rigorous novelty – so I would encourage open peer review as a way to achieve that. Not sure it can be done with metrics (a focus on counting citations doesn’t do it).
We want to reward real-world impact, where that arises (while also acknowledging that it won’t happen immediately or in all cases). But as we know from the REF, impact is very hard to capture with metrics.
Final challenge: We also need to remember the people – the researchers. Too much focus on counting outputs and inputs makes us liable to forget about the importance of collaboration, about team-players (in research groups AND in the teams needed to keep depts running); and to forget about quality of life issues (among which I would include equality, diversity and inclusion). There are some indicators that can capture aspects of quality of life and EDI but I don’t see them being used widely enough.

To what extent are these challenges global and to what extent local?

They are both.
As Assistant Provost for Equality, Diversity and Inclusion (EDI) at Imperial College I am mindful of the significant variation in attitudes to all sorts of topics (including EDI but also research assessment) within an institution – between different depts, never mind between institutions or different countries…
Culture is very granular – and the only way to explore it is by talking to people. As Atul Gawande wrote:“People yearn for frictionless technical solutions, but people talking to people is still the way the world’s standards and norms change.” So local discussions with researchers are vital.
But on their own they are not enough. It doesn’t matter if only a handful of organisations, stimulated by DORA or the Leiden Manifesto or whatever, establish a principled stand on responsible metrics and develop robust, holistic and evidence-based procedures for research assessment that are free from bias.This has to be an international project because research and researchers are internationally mobile.

Where might an international approach help solve these challenges?

As Chair of DORA, obviously I think the organisation has a key role to play. We are the only independent, international organisation that is focused on the reform of research assessment and that is actively campaigning for change.
We are doing lots of things:
- promoting the idea and recruiting signatories
- collecting and disseminating good practice
- running workshops to critique and develop good practice in reseach assessment
- established an international advisory board
But there is lots more that we can do, and that we would like to do;
- critically evaluate what works (we need to provide tools that work or people will fall back on the JIF)
- build an international alliance and presence in all parts of the world
- reach out more to arts, humanities and social science scholars

How can Bibliometric practitioners work together to help achieve this?

Work with researchers so that you really understand the constraints that they work under. Remember Gawande…
Health warnings on the ever-present limitations of indicators: The statement “complex human activity cannot be boiled down to numbers” should be attached to every paper on bibliometrics, every bibliometrics database, and every league table…
Not sure if this is your bag but: Be louder in your criticisms of university league tables that aggregate scores into a single number; as presently constructed, most tables are deeply flawed. Anyone who celebrates their institutional ranking is admitting to an alarming degree of intellectual vacuity.
Do more to examine and expose the biases present in the numbers.
Work with DORA!

A final message for all of us:

Let’s get away from the “dodge culture”. We’re all good at signing up to principles and describing the problem. But we are also all good at wringing our hands and dodging the consequences, so that we don’t have to think constructively about workable solutions. I’ve seen that in the debates on Plan S (which has a strong research assessment component) – there are lots of complaints but not nearly enough constructive engagement.

But I’ve gone on for long enough so we can talk about that if it comes up in the questions…

Posted in Science | Comments Off

Endings and Beginnings

Posted on December 29, 2018 by Stephen

New Year’s Eve is almost upon us, so here we are again at the close of one long year and the start of another. Personally, it has been a year of endings and beginnings. Readers of this blog would be forgiven for thinking that it is one of the things that I have wound down in 2018, but in fact I am hoping to stir it to new life.

Sunrise or sunset? Black Sea coast near Sozopol.

The things that have ended for me this year include Science is Vital, which closed its doors in the autumn because the busyness of the lives of too many of us on the executive committee made the commitment to the organisation impossible to sustain. We shared a certain sense of satisfaction at having helped to move the debate about public investment in research and development to a point where all three political parties made substantial commitments in the general election of 2017, which, for its part, the present government is attempting to deliver. But the darkness of uncertainty in which Brexit has shrouded the future of the UK made it an uneasy termination.

My six-year term as a member of the board of the Campaign for Science and Engineering (CaSE) also came to an end. CaSE, established as Save British Science in 1986, was very much a forerunner to Science is Vital though it has matured professionally over the decades. It was a privilege to play a small part in supporting the organisation and to see it grow more prominent and influential than ever in the UK science policy landscape, thanks in no small part to the sure-footed leadership of Executive Director Dr Sarah Main and Chair Professor Graeme Reid. I will continue to support CaSE as a member and would encourage you (and your university, if you work at one) to do the same.

In the summer the Guardian closed the Science Blog Network where I had had been writing, as part of the Occam’s Corner crew, since 2012. It had been from the very start a bold experiment, since the Guardian effectively gave the keys to their publishing platform to a bunch of scientists and science historians and, apart from asking us to check any potentially libellous materials their legal team, left us to our own devices. But financial pressures eventually came to bear. In the last couple of years we were increasingly expected to achieve a minimum number of page-view with each post. That was hardly an unreasonable demand given the Guardian’s dependency on advertising income but it did curtail the freedom to explore some of the more niche areas, which seemed to be part of the point of starting a blog network in the first place. I guess those financial pressures became overwhelming this year. In my own case I can’t entirely blame the Guardian (though it would have been nice to have a clearer explanation of their decision), since changes in my day job had reduced my output to a trickle. I’ll always be grateful for an opportunity that was both terrifying and empowering.

That change in my day-job was becoming Assistant Provost for Equality, Diversity and Inclusion (EDI) at Imperial College. Strictly, this new role had started in October 2017 but 2018 was the year in which it really began to take shape. After an extensive consultation, drafting and re-drafting process involving a great many people, I published the university’s first EDI Strategy. This has a substantial action plan attached which means that we have set quite a few balls rolling, including revision of our policies on sexual harassment, signing up to the Race Equality Charter, re-vamping our approach to supporting LGBTQ+ staff and students, and trying to ensure that support for disabled staff and students is as consistent and mainstreamed as possible. These are still rolling and several of them threatened to outrun me. In combination with teaching and departmental duties, this made the final third of the year a lot more stressful than I would have liked.

Stress is life’s way of telling you you’ve reached your limits. You might think someone well into middle age would know his limits, but I’m still triangulating. The problem is partly that the challenges of EDI are numerous and incompletely defined – we are talking about culture change after all, and the dismantling of often contentious barriers to inclusion. But it also arise partly because I have an over-developed tendency to under-estimate how long it takes to get things done. For the sake of my sanity I aim (with the help of some management coaching) to discipline myself and my work-schedule for the coming year. But even as I write this I can hear the faint sound of laughter in the distant reaches of my mind. Wish me luck.

The final new beginning for me in 2018 was becoming chair of the steering group of the San Francisco Declaration on Research Assessment (DORA), a six-year-old initiative aiming to eliminate the dependency on crude metrics that has taken hold within academia. DORA, which has long been close to my heart, was granted a new lease of life this year thanks to support from a number of funders and publishers. We launched a new web-site in February, published a roadmap in June outlining our immediate priorities, and convened the first meeting of our new international advisory board in October. The initiative is gathering momentum alongside other moves to reform research evaluation by funders, learned societies, and research institutions (many of them highlighted on the DORA Good Practices page).

DORA got a significant boost in the announcement of Plan S, the radical and controversial European project to accelerate the uptake of open access publishing. Plan S recognises that academic obsession with journal prestige is an impediment to the development of cost-effective open access outlets for publicly-funded scholarship and is insisting that implementation of DORA or DORA-like principles has to go hand in hand with publishing innovations designed to increase the accessibility of new research findings. Much discussed and argued about here and elsewhere (and currently under consultation), you can expect Plan S to feature on this blog in 2019, though I will be treading a careful line, since DORA is a wholly independent organisation and has no official position on the initiative.

I can still feel the grip of these endings and beginnings. There is more to say on each but that will do for now. As well as being a pause for thought at the year’s end, this post was an attempt to get a writing muscle that had been idle too long back into action. The idea of the blog being a place to think out loud in public retains it appeal and although this one had been in danger of falling dormant, I hope to clear the time in 2019 for a re-awakening and a new beginning.

Assuming of course that the dreadful mess that is Brexit doesn’t become all-consuming.

Wishing you all a happy and hopeful new year.

Posted in Blogging, Scientific Life | 4 Comments

Academic freedom and responsibility: why Plan S is not unethical

Posted on October 1, 2018 by Stephen

Since its announcement on 4th September the European Commission’s plan to make a radical shift towards open access (OA) has caused quite a stir. Backed by eleven* national funding agencies, the plan aims to make the research that they support free to read as soon as it is published. This is a major challenge to the status quo, since the funders are effectively placing subscription journals off limits for their researchers, even if the journals allow green OA (publication of the author-accepted manuscript) after an embargo period; Plan S also specifically excludes hybrid open access except in cases where journals have an agreed schedule for flipping to OA. The plan has been welcomed as “admirably strong” by OA advocate Peter Suber, though he has also offered cautionary notes on some aspects. Others have been less enthusiastic. A central charge, from some publishers and some academics is that Plan S is an infringement of academic freedom to choose how and where your work is published and is therefore unethical.

I disagree. The claim that Plan S is unethical derives from an understanding of academic freedom that appears to me to rest on foundations that, if not shaky, are at least highly questionable.

I realise this is contentious territory. And I accept that the views of many scholars on this issue spring from deep convictions about the nature of the academic calling. There are real complications and tensions, not the least of which are disciplinary differences, especially between the natural sciences, engineering and medicine on the one hand, and the arts and humanities on the other. I therefore propose to tread carefully. But ultimately I believe the disputes about Plan S can be located at the blurry, shifting and negotiable boundary between academic freedom and responsibility and it is this boundary that I am mainly interested in exploring.

Let’s look first at the particular claims made with regard to the incursion of Plan S into academic freedom.

The statement made by the International Association of Scientific, Technical and Medical Publishers (STM) asserts that “it is vital that researchers have the freedom to publish in the publication of their choice”. No documentary support is offered to justify its interpretation of academic freedom so we must look elsewhere. Before doing so we should note that the STM’s view fits well with the status quo – a state of affairs that is financially very rewarding for many publishers, so there is a vested interest in play that must inevitably colour any assessment of their claims.

The academic view advanced in trenchant terms by Kamerlin and colleagues is that “Plan S clearly violates one of the basic tenets of academic freedom – the freedom to publish research results in venues of the researcher’s choosing.” However, despite the fact that the authors of this piece are all researchers or scholars of one type or another, they give no citations to back up their claim. Perhaps they take that feature of academic freedom as a given, something that is so widely accepted that it no longer requires a supporting reference. If so, that is a difficult position to sustain. For one thing, choice of publishing venue is not mentioned at all in the community-authored Wikipedia article on academic freedom. If we are to properly debate the question of whether choice of publication venue is a “basic tenet” of academic freedom, we need an evidence base of some sort.

J Britt Holbrook, one of Kamerlin’s co-authors, goes into more depth in a response to Marc Schiltz, the president of Science Europe and one of the prime movers behind Plan S. Holbrook mentions the definition offered by the American Association of University Professors in their 1940 Statement of Principles on Academic Freedom and Tenure, which presumably is widely accepted by scholars in the USA, though I don’t know how much purchase this document has in the rest of the world. The AAUP defines three elements to academic freedom, the first of which pertains to the conduct of research:

“Teachers are entitled to full freedom in research and in the publication of the results, subject to the adequate performance of their other academic duties; but research for pecuniary return should be based upon an understanding with the authorities of the institution.”

The key phrase is in the first sentence – “full freedom in research and in the publication of results”. To some that implies full freedom in the choice of where to publish. As I understand Holbrook, this is the interpretation that he places on it. But the document isn’t specific on the point and other interpretations appear equally valid. The intent could be merely to guarantee the right to publish, free from any political or institutional constraint on what you can write, without implying that authors must also be completely free in the choice of venue. Indeed, on this point it is important to note the second, qualifying part of the sentence – “subject to the adequate performance of their other academic duties” – which raises the question of academic responsibilities, at least to their university.

The latter part of that quotation from the AAUP statement is perhaps even more relevant to Plan S (which, as I understand it, applies only to funded research). The AAUP notes that “research for pecuniary return” must be subject to an “understanding” with the employing university. The phrase “pecuniary return” is open to various interpretations. It could mean a direct fee, or grant funding that contributes directly or indirectly to the academic’s salary. The AAUP is rather vague on the point, but this qualifier does seem to suggest that academic freedom is subject to consideration of the interests of other stakeholders.

This isn’t the only instance of vagueness in that short sentence. Consider what exactly the AAUP means by “full freedom in research”. Clearly, in part, it seems reasonable to interpret this as asserting freedom to research any question or topic. But what about academic freedom in their tools of enquiry? In the UK, for example, when funding is awarded to researchers for the purchase of large items of equipment, there is usually a requirement to ask for price quotations from two or more suppliers and an accompanying duty to secure good value for money in the purchase. On a narrow interpretation of the AAUP statement, this would constitute an infringement of academic freedom, though I have never heard of anyone making such a complaint. While this may seem like a trivial consideration in the present discussion, to me it illustrates the complexity underlying even apparently simple statements. Here it is a reminder of the need to be very careful of context when examining the justice of particular claims made for academic freedom.

Where else might we look for greater clarity on the shared understanding of academic freedom? One of the more detailed documents that discusses and defines academic freedom is UNESCO’s Recommendation concerning the Status of Higher-Education Teaching Personnel, published in 1997 and first brought to my attention earlier this year on Twitter by the journalist Richard Poynder. This is a long and detailed document but in paragraph 12, highlighted by Poynder, there appears to be an unequivocal assertion that the definition of academic freedom includes choice of publishing venue:

“higher-education teaching personnel should be free to publish the results of research and scholarship in books, journals and databases of their own choice and under their own names, provided they are the authors or co-authors of the above scholarly works.”

However, again, it is important to examine the context of such statements and here at least the UNESCO document is more helpful than the much briefer AAUP statement. UNESCO provides a lengthy preamble to locate its definition of academic freedom in appropriate social, political and academic contexts. The preamble reflects not just the rights of academics but also their duties, and the rights and expectations of other stakeholders. In composing the document, the authors are “conscious that higher education and research are instrumental in the pursuit, advancement and transfer of knowledge” and “conscious that governments and important social groups, such as students, industry and labour, are vitally interested in and benefit from the services and outputs of the higher education systems”; they are concerned “regarding the vulnerability of the academic community to untoward political pressures which could undermine academic freedom”; and mindful that “the right to education, teaching and research can only be fully enjoyed in an atmosphere of academic freedom and autonomy for institutions of higher education and that the open communication of findings, hypotheses and opinions lies at the very heart of higher education.” (Italicised emphasis above and elsewhere added by me).

In outlining the guiding principles of the UNSECO document, the issue of academic responsibility is fleshed out in more detail. Its authors note the need to ensure that academics are free to “pursue new knowledge without constriction” but also note that “teaching in higher education is a profession: it is a form of public service that requires of higher education personnel expert knowledge and specialized skills acquired and maintained through rigorous and lifelong study and research; it also calls for a sense of personal and institutional responsibility for the education and welfare of students and of the community at large.” They also observe that “where public funds are appropriated for higher education institutions, such funds are treated as a public investment, subject to effective public accountability”, that “the funding of higher education is treated as a form of public investment the returns on which are, for the most part, necessarily long term, subject to government and public priorities” and that “the justification for public funding is held constantly before public opinion.”

These statements come just before paragraph 12 (quoted in part above) and must be borne in mind if we are to better understand the authors’ intentions. To help with that, it is worth also considering paragraph 12 in full:

“12. The publication and dissemination of the research results obtained by higher-education teaching personnel should be encouraged and facilitatedwith a view to assisting them to acquire the reputation which they merit, as well as with a view to promoting the advancement of science, technology, education and culture generally. To this end, higher-education teaching personnel should be free to publish the results of research and scholarship in books, journals and databases of their own choice and under their own names, provided they are the authors or co-authors of the above scholarly works. The intellectual property of higher-education teaching personnel should benefit from appropriate legal protection, and in particular the protection afforded by national and international copyright law.”

The preamble and principles put a clear emphasis on academic freedom as a freedom from undue political interference in the questions that academics may ask and write about, and it is this concern that seems uppermost in their minds when they write about the freedom to publish. One interpretation of their specific statement about journal choice could therefore be that they wanted academics not to be forced by instructions from governments or institutional leaders to only publish in venues that might limit the exposure of their ideas. There is a clear emphasis also in the document on public service and accountability, which envisages academic scholarship and professionalism as things that are valued certainly, but also as things in which the public has a vested interest. Might that interest also reasonably include a desire to facilitate the widest possible dissemination of research and scholarship?

Further uncertainty about the precise intent of the UNESCO authors with regard to mentioning choice of publication venue in their definition of academic freedom comes from their restatement of the definition in paragraph 27 as:

“the right, without constriction by prescribed doctrine, to freedom of teaching and discussion, freedom in carrying out research and disseminating and publishing the results thereof, freedom to express freely their opinion about the institution or system in which they work, freedom from institutional censorship and freedom to participate in professional or representative academic bodies.”

In this formulation there is no specific mention of choice of where to publish. If, as claimed by Kamerlin and colleagues, this choice is a “basic tenet” of academic freedom, one would expect to see it stated consistently in authoritative documents.

I would argue that neither the AAUP statement nor the UNESCO document posits the free choice of where to publish academic work as a core component of academic freedom or an unfettered right. The various statements, particularly in the UNESCO document, about the wider public interest in academic research and teaching support the view that academic freedom is understood to operate in an environment where a reasonable level of academic responsibility and accountability are also expected.

Clearly there are complications here, but I hope at least that the above analysis gives a clearer view of the boundary where Plan S has landed. For what it’s worth I believe that academics should certainly be as free as possible to choose where to publish, in acknowledgement of their professionalism and expertise. I think it is therefore important that the implementation of Plan S strives to ensure that there remains a rich variety of outlets. But we also need to acknowledge that at present academics’ publishing choices are constrained by the perverse incentives that have grown up around metrics of journal prestige. For that reason, I was pleased to see that reform of research evaluation is at the heart of Plan S. If it can help to drive real change on this front, arguably Plan S will make a positive contribution to academic freedom.

Clearly also, the questions of how different academics envisage their responsibilities and what might constitute a reasonable level of public accountability remain valid matters for debate. One also has to remember that the UNESCO document was written in 1997, well before the Berlin Declaration on Open Access which came along six years later. It seems highly unlikely that the UNSECO authors would have given much consideration to the implications of the impact of open access on academic freedom. That said, paragraph 13 hints that the authors might have seen open access, an idea that owes its being to the digital technology of the internet, as an opportunity to enhance academic freedom by facilitating the exchange of ideas:

“13. The interplay of ideas and information among higher-education teaching personnel throughout the world is vital to the healthy development of higher education and research and should be actively promoted. To this end higher-education teaching personnel should be enabled throughout their careers to participate in international gatherings on higher education or research, to travel abroad without political restrictions and to use the Internet or video-conferencing for these purposes.”

Of course, we cannot know what their view might have been of how open access or Plan S interact with academic freedom. Times have moved on and the UNESCO document is now over twenty years old. But these are matters that have to be kept under continuous review and debate as technology and norms change. One point where I am in complete agreement with Kamerlin and her co-authors is that the voices of academics certainly have to be heard as Plan S moves to implementation and I understand that moves to facilitate that dialogue are already afoot.

Part of that dialogue will also need to consider the question of licencing under Plan S which is need of clarification. If I have read carefully, a significant portion of Holbrook’s claim that Plan S is unethical is related to the plan’s explicit and implicit requirement for a CC-BY licence on published research, which permits access and re-use, in whole or in part, by commercial and non-commercial users as long as proper attribution is made to the original authors. While it should be emphasised that Plan S also envisages authors retaining copyright on their publications (perhaps along the lines developed by the UK Scholarly Communications Licence), there is resistance to the use of CC-BY licences, particularly among some humanities scholars (though not all) and also among some researchers in the natural sciences.

I am not going to dwell on the details of this debate except to observe that humanities scholars appear to be more personally invested in their published works (in addition to often not being directly funded to write them, though the question of funding raises yet further complications) and that that inevitably affects their view of their rights as authors. Many of these issues have been explored in depth by Martin Eve in his freely available book, Open Access and the Humanities. I will focus instead on the perceptions of research scientists, largely informed by my own experiences.

I think it is fair to day that there is still a lot of confusion among academics about what licensing means. Like many other scientists, for most of my career I never gave copyright a second thought and willingly – though not really knowingly – transferred it for free entirely and exclusively to the publishers of my papers. But I’m thinking about it now thanks to the debates aroused by open access about what scholarly publishing is for. And I find that I can’t think about it properly without also taking account of my rights and responsibilities as a researcher and an academic. Some of those responsibilities arise from the contracts that I entered into when I accepted funding for my research – typically from a publicly-funded UK research council or a charity such as the Wellcome Trust. In the same way that I didn’t see requirements to submit equipment purchases to a process of competitive tender as unreasonable, I haven’t seen requirements that seek to maximise the accessibility of my published results as undue interference in my research or an infringement of my academic freedom. This is because I accept there has to be a reasonable balance between my freedom to explore the questions of my choosing, and the funders’ interest in the return on their investment.

Of course, there is a proper debate to be had about what funders should consider to be a reasonable return. The arrival of Plan S is just the latest opportunity to revisit it. At present in the UK it seems to me we have – give or take the occasional protest – a shared recognition among politicians, academics and the public that there needs to be a balance between blue skies, curiosity-led research and work that is more strategic or applied. In an open society, the particular point of balance is rightly a matter of public debate between these different stakeholders. In Britain in recent years much of this discussion has revolved around the tensions arising from the impact agenda, a topic that has been explored in an interesting and constructive article on academic freedom published by Holbrook in 2017. Holbrook raises important points about the neoliberal capture of science policy via the impact agenda (particularly by the EU), though in my view overplays the influence of that agenda and underplays the extent to which seeking real-world impact from research work is in tune with the motivations that bring many scientists to the laboratory in the first place. Nevertheless, he concludes that academics need both to develop a more positive view of academic freedom that embraces the common good and to engage in conversation with “the communities in which our universities are situated”.

On that last point we can certainly agree. I look forward to the continuing conversations – with other academics and the communities we serve – about Plan S and about how best to define and protect academic freedom.

*Update 01 Oct 2018, 10:23 – there are now twelve funders backing Plan S. Thanks to Ross Mounce for the correction.

Posted in Open Access | Tagged Academic Freedom, open access, Plan S | 49 Comments

Ten Years a Blogger

Posted on September 9, 2018 by Stephen

Today is the tenth anniversary of my very first blog post. When I look back at that day in 2008 when I set out my stall on Reciprocal Space it seems a long time ago and a long distance away. It’s been quite a journey.

Some things haven’t changed. I still hate the terminology, though I have mostly managed to swallow my embarrassment.

“I know the etymology: web-log, b-log, blog. It makes perfect sense but it’s such a silly-sounding word that it seems to demean the process. I would be embarrassed to admit that I do it, just because it has such a stupid name.”

And my blog manifesto is unaltered, though I like to think that my writing has on occasion given readers pause for reflection.

“I won’t promise to post regularly; that way I will avoid the repetition of future apologies for failing to write. I won’t promise to be unembarrassed to admit that I am a blogger. I won’t promise to have anything terribly insightful to say.”

Starting a blog has had many unintended but interesting consequences. I enjoyed the free rein of it, the chance to write about whatever liked – books I’d read, nights at the synchroton, time served on grant panels and, of course, impact factors.

I launched Reciprocal Space not long after becoming a professor but blogging led me as never before to reflect on the business of research, the internal culture of academia and the interactions of scientists with the rest of society. It got me involved in Science is Vital’s impassioned and effective campaign for science funding in the UK; in debates about open access and scholarly publishing (still very much ongoing and in the news this week with the announcement of European Commission’s radical Plan S); in policy work on research evaluation – I co-authored the Metric Tide report and now have the honour of serving as Chair of the San Francisco Declaration on Research Assessment (DORA); and even, sporadically, in film-making.

As part of the Occam’s Typewriter crew I also had the privilege of writing for the Guardian Science Blog network for the past six years. That was exciting and terrifying in equal measure but allowed us to reach entirely new audiences. Sadly the Guardian decided to close the network at the end of August. None of us involved are quite sure why. Perhaps it wasn’t generating the traffic needed to be sustainable. Perhaps their priorities simply shifted. But I am sorry to see it go, not only for the loss of a prominent platform, but because I think that the bloggers there were often able to provide a reasoned counter to some of the wilder claims made daily in politics and the media. I think I had always tried to take the wider, more understanding vew – even of Brexit (though in 2016 I severely over-estimated the British government’s capacity for pragmatism) and the opinions on science of Simon Jenkins.

But times change. The blogosphere itself has altered beyond recognition from the heady days of 2008. Much of the commenting activity has migrated to Twitter, whose short form seems only to have intensified the vitriolic tendencies of discussion on social media. And nothing lasts for ever, so the termination of the Guardian Science Blog network is chance to think anew. Several new portals have already opened – at the new Cosmic Shambles blog or on *Research. Others are currently under discussion.

For myself, I’m currently considering options. I hope at least to contribute to the Scientists for EU blog; as far as am concerned Brexit remains the most serious threat to the polity and prosperity of this country.

I am also trying to recalibrate how I can balance writing with my responsibilities at Imperial, where I have taken on the role of Assistant Provost for Equality, Diversity and Inclusion (in which capacity I recently started another blog!). Transitions and balances can be tricky but I sense an opportunity and, though ten years older, I feel strangely energised for a new challenge.

A chapter may have closed but the book is still open.

Posted in Blogging, Communication, Scientific Life | 4 Comments

DORA, the Leiden Manifesto & a university’s right to choose: a comment

Posted on July 18, 2018 by Stephen

The post below was written as a comment on Lizzie Gadd’s recent post explaining in some detail Loughborough University decision to base their approach to research assessment more on the Leiden Manifesto than DORA, the Declaration on Research Assessment. So you should read that first! (The comment is currently ‘in moderation’ because, like myself, Lizzie is on holiday. I suspect she is more disciplined that I am at not looking at her email whilst on holiday. I’ll update this post once the comment is approved.

Update (18-07-18, 15:30) – the comment has now been approved. I suggest any further discussion takes place beneath Lizzie’s original post.

Even though I am currently chair of the DORA steering committee, I don’t want to get into ‘theological’ arguments about the differences between DORA and the Leiden Manifesto because they are both forces for good! Moreover, I am sure that Lizzie agrees with me that ultimately it is the development of good research assessment practices that matter and I again applaud the work that she has been doing on that front at Loughborough.

Nevertheless, I want to argue for a more expansive interpretation of what DORA means (as a declaration and an organisation) than is presented here.

It is perfectly true that DORA was born in 2012 but it would not be correct to suppose that the declarationis any more fixed in time than the Leiden Manifesto. Although the first and most prominent recommendation of the declaration is stated negatively (“Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.”), the remaining 17 recommendations are almost invariably positive, encouraging adherents to think about and create good practice. It does not limit how they should do that. The declaration is not very long so I would encourage everyone to read it in full.

Nor should it be supposed that DORA’s relevance is confined to the sciences; it has always aimedto be “a worldwide initiative covering all scholarly disciplines”. Admittedly, the work to extend that coverage has been lacking, but as the recently published roadmap makes clear, we have now placed a particular emphasis on extending DORA’s disciplinary and geographical reach. We are in the process of assembling an international advisory board from all corners of the globe. We would be glad to hear from arts, humanities and social science scholars about how DORA can help them to promote responsible research assessment in their fields.

Lizzie draws a careful distinction between not using journal metrics to assess the quality of research outputs and using them to assess ‘visibility’. She is right to do so because there is a risk it might send a subliminal message to researchers. It would be interesting to hear from Loughborough’s researchers how they interpret the guidance on these points.

This distinction is the basis of Lizzie’s argument that, because Loughborough wishes to incentivise it’s researchers to make their outputs more visible, they could not in good conscience sign DORA. I can see how that is an honest interpretation of the constraints of the declaration, but my own view is that it is too narrow. The preamble to the declaration lays out the argument for the need to improve the assessment of research ‘on its own merits’. This and the thrust imparted by the particular recommendations of the declaration show that it is the misuse of journal metrics in assessing research content – not its visibility – that is the heart of the matter. It seems to me that Loughborough’s responsible metrics policy is therefore not in contravention of either the letter or the spirit of DORA.

In the end, as Lizzie rightly states, it is Loughborough’s call and, again, I am sure that Lizzie and I have in common a strong desire to promote good research assessment practices. I stand by what I wrote back in 2016, in a piece bemoaning the fact that so few UK universities had yet to sign DORA:

“I would be happy for universities not to sign, as long as they are prepared to state their reasons publicly. They could explain, for instance, how their protocols for research assessment and for valuing their staff are superior to the minimal requirements of DORA. It’s the least we should expect of institutions that are ambitious to demonstrate real leadership.”

Posted in Academic publishing | Comments Off

Ready-made citation distributions are a boost for responsible research assessment

Posted on July 1, 2018 by Stephen

Though a long-time critic of journal impact factors (JIFs), I was delighted when the latest batch was released by Clarivate last week.

It’s not the JIFs themselves that I was glad to see (still alas quoted to a ridiculous level of precision). Rather it was the fact that Clarivate is now also making available the journal citation distributions on which they are based. This is a huge boost to the proposal made a couple of years ago by myself, Vincent Larivière and several prominent editors and publishers that citation distributions should be published by journals that advertise their JIFs.

Our proposal aimed to reduce the persistent and highly toxic influence of the unidimensional JIF on research assessment by giving authors and readers had a much richer picture of the real variation of citation performance within any given journal. It depended for its impact on journals following the recipe we provided in our preprint for generating the distributions from proprietary citation data in Web of Science or Scopus. Although a number of enlightened editors were quick to adopt the practice (e.g. at PNAS, Acta Cryst. A), it did not spread as far or as rapidly as we woudl have wished. However, now that the distributions are available ready-made from Clarivate*, there is no reason for any journal not to follow suit.

I hope that many will now opt to do so. Bianca Kramer (aka @MsPhelps) was quick off the mark publishing a couple of examples of citation distributions from well-known journals (see above). The enormous range of citation performance is immediately apparent.

Commendably, Clarivate have gone further and disaggregated research papers from reviews and other article types in the distributions. This is something the San Francisco Declaration on Research Assessment has long called for (see point 14 in the text of the declaration); it helps to further untangle the complexity underlying the JIF. Helpfully, Clarivate also now separately report the median citation rates of primary research articles and reviews. This reveals how highly-cited reviews inflate the indicator.

So I congratulate Clarivate on a positive move that provides the information needed for more responsible use of quantitative publication indicators. Of course, the goal of establishing robust research assessment processes that are free of the JIF has yet to be achieved. We need all journals that make any mention of their JIFs to also show Clarivate’s citation distributions. And we need researchers and research managers to start internalizing what they mean.

Even then there will be more to do. For one, Elsevier has recently made a big move into journal metrics with the publication of CiteScore, its alternative to the JIF; I hope that they too will start making the underlying citation distributions available.

Then there is the residual problem of that other unidimensional indicator, the h-index. In my view, any researcher who quotes their h-index should be expected to also show the underlying citation distribution. I have done this myself recently (using data gathered manually from Google Scholar – see above) as a way of showing that there are interesting and impactful stories to tell about my research papers irrespective of where they appear on the citation distribution.

Numbers are too powerful sometimes. In assessing any complex human activity like research, it is the stories, the narratives that provide the context essential to making an informed judgement.

*Important note about the use of Clarivate’s citation distributions: The initial press release didn’t mention the re-use rights are for the plots that subscribers can now find on Web of Science but Clarivate’s Marie McVeigh has now clarified this officially in the comment below. She confirms that journals can indeed publish these distributions (with attribution).

**For those familiar with R, Steve Royle has published a clever script that makes it easy to grab and h-index citation data from Google Scholar.

Posted in Academic publishing | 1 Comment

Opening peer review for inspection and improvement

Posted on February 14, 2018 by Stephen

For me the most memorable event at last week’s ASAPbio-HHMI-Wellcome meeting on Peer Review, which took place at HHMI’s beautifully appointed headquarters on the outskirts of Washington DC, was losing a $100 bet to Mike Eisen. Who would have guessed he’d know more than I did about the intergalactic space lord and UK parliamentary candidate, Lord Buckethead? Not me, it turned out.

Mike was gracious enough not to take my cash, though now I owe him multiple beers over the coming years. I doubt that the research community will have fixed peer review within that timeframe – there are no one-size-fits-all solutions to the multiple problems that were discussed at the meeting – but I did at least come away with a sense that some real improvements can be made in the near future. The gathering was a heady mixture of expertise, passion, ambition, blue-skies thinking and grey-skies reality checks.

I won’t attempt to summarise all of our deliberations. My brain no longer works that way and in any case the super-capable Hilda Bastian did it as we went along; (and there is also a brief report in Science). Hilda was one of several people I knew from the internet whom I met in the flesh for the first time in Washington. For all our technology, nothing can yet touch the levels of sympathetic engagement that comes from encountering your peers in the real world. It is still the best place for exchanging ideas — and for (ahem) exposing them to sobering correction.

After all was said and done, there were three ideas that I hope will endure and soon become more widely adopted. Each carries a modest cost but the benefits seems to me to be incontrovertible.

1. Open Peer Review: There are many definitions of ‘open peer review’ (at least 22 according to Tony Ross-Hellauer), but the one I have in mind here is the practice of publishing the reviewers’ anonymous comments, alongside the authors’ rebuttal, once a paper is finally accepted for publication. To my mind this degree of openness improves peer review because it incentivises professional conduct on the part of reviewers, thereby reducing the scope for sloppy work or personal attacks. It also makes the conduct of science significantly more transparent. While there was some concern that special interest groups operating in contentious areas of research like vaccines and climate science might derive ammunition by cherry-picking critical reviews, I am firmly of the view that the research community has to be ready to defend itself in the open. Closing the door on our proceedings and expecting the public to trust is will only fuel those who are already too quick to malign the scientific establishment as a conspiracy.

There are yet more benefits. Opening peer review helps to reveal the human and disputatious nature of progress in research. That will dispel the gleam if objective purity that sometimes clings, unhealthily, to the scientific enterprise. Being more open about how science works will build rather than undermine public trust. It will also help to burnish the reputation of journals that insist on rigorous peer review, and expose the so-called predatory journals that levy hefty article processing charges while providing no meaningful review.

Open peer review also paves the way for reviewers to claim credit for their work. This can be done anonymously via systems like Publons, which liaises with journals to validate the work of reviewers. Greater credit can of course be claimed if the reviewer agrees to sign their review, since this allows them to be cited and accessed more effectively (especially if they are assigned a DOI – digital object identifier). However, views at the meeting on whether reviewers’ names should be made public were mixed. There are concerns that early (and not-so-early) career researchers might pull their punches if reviewing the work of more senior people. And there are risks, as yet untested as far as I know, of possible legal action against reviewers who make mistakes or who criticise the work of litigious researchers or corporations.

The overheads associated with publishing reviews are not negligible. Some effort will be required to collate and edit reviews (e.g. to remove personal or legally questionable statements, though this should reduce as reviewers become accustomed to openness); and there are technical hurdles to the inclusion of publishing reviews in journal workflows. However, none of these is insurmountable, since several journals (e.g. EMBO Journal, PeerJ, Nature Communications) are already offering open peer review.

As Ron Vale wrapped up proceedings on the main day of the conference, I could sense him urging the room with every fibre of his being to recognise open peer review as an idea whose time has come. I think he’s right.

2. Proper acknowledgement of peer reviews by early career researchers (ECRs): Although the vast majority of peer review requests are issued to established researchers, in many cases the work is farmed out to PhD students and postdocs. When done properly, this can provide valuable opportunities for ECRs to learn one of the important skills of the trade, but too often their contributions are unacknowledged. Either the principal investigator does not bother to explain to the journal that the review is entirely or partially the work of junior members of their lab, or even if they do, the journal has no mechanism for logging or crediting that input.

It became clear at the meeting that this is an issue for ECRs and a sore one at that. Ideally of course, the fix would come from PIs being more transparent about how they handle their reviewing caseload, but a more effective solution would seem to be for journals to issue clearer guidelines – both to enable PIs to recruit ECRs to the task and to ensure that the journal formally recognises their contribution. Services such as Publons that offer to validate peer review contributions could also help out here.

3. Add subjective comments to ‘objective’ peer review: This is a counter-intuitive one. I am a fan of the ‘objective’ peer review established at open access mega-journals such was PLoS ONE, where the task of the reviewer is to determine that the work reported in the submitted manuscript has been performed competently and is reported unambiguously, but without any regard for its scientific significance or impact. But, as was pointed out in one of the workshops at the meeting, reviews adhering to these criteria may well be devoid of the full richness that the expert reviewer has brought to their close reading of the manuscript. It was therefore proposed (though Chatham House rules prevent me from crediting the proposer) that this expert opinion should be added to the written review. It should not form any part fo the decision to publish but inclusion of this additional information – on the significance of the study and the audiences that would be most interested in it, for example – would provide a valuable additional layer of curation and filtering for the reader.

This proposal may be tricky to implement because the editors of mega-journals already have enough trouble getting some reviewers to stick to the editorial standard of ‘objective’ peer review. But it does not seem impossible to add a separate comment box to the review for to ask the reviewer’s opinion – as a service to the reader – while making it clear that this will have no impact on the publishing decision. To me this is only a small additional ask of the reviewer, but one whose value is obvious. Such a move would also be a further barrier to the rise of the so-called OA predatory journals.

And finally… it would be remiss of me not to mention DORA, the San Francisco Declaration on Research Assessment which is endeavouring to wean the research community from the nefarious effects of journal impact factors. While the meeting was focused on peer review of research papers, it is also an important component of decisions about hiring, promotion and grant funding. The new steering group of DORA, of which I am now chair, were grateful for the opportunity to announce the reinvigoraton of the initiative and to discuss how DORA might help the community to develop more robust methods for research and researcher assessment.

And that’s it. Of course, you should feel free to offer peer review in the comments…

Posted in Academic publishing, Science | Tagged asapbio, hhmi, peer review, Wellcome Trust | 1 Comment

Berlin Debate – Who owns science?

First impressions: the DORA-HHMI meeting on research assessment reform

I told myself…

The unsustainable goal of university ranking

Who’s the best? (Original photos via Wikipedia – click image to see links)

How to value what cannot be measured?

Thinking globally about research evaluation – LIS-Bibliometrics talk

In your view, what are the main research evaluation challenges we face?

To what extent are these challenges global and to what extent local?

Where might an international approach help solve these challenges?

How can Bibliometric practitioners work together to help achieve this?

Endings and Beginnings

Academic freedom and responsibility: why Plan S is not unethical

*Update 01 Oct 2018, 10:23 – there are now twelve funders backing Plan S. Thanks to Ross Mounce for the correction.

Ten Years a Blogger

DORA, the Leiden Manifesto & a university’s right to choose: a comment

Ready-made citation distributions are a boost for responsible research assessment

Opening peer review for inspection and improvement

Recent Posts

Recent Comments

Archives

Categories

Blogroll

Meta