I came across a particularly dispiriting example of this recently when I was asked by a well-known university in North America to help assess the promotion application of one of their junior faculty. This was someone whose work I knew — and thought well of — so I was happy to agree. However, when the paperwork arrived I was disappointed to read the following statement the description of their evaluation procedures:

“Some faculty prefer to publish less frequently and publish in higher impact journals. For this reason, the Adjudicating Committee will consider the quality of the journals in which the Candidate has published and give greater weight to papers published in first rate journals.”

Which means of course that they put significant weight on impact factors when assessing their staff. Given the position I had developed in public (and at some length) I felt that this would make it difficult for me to participate. I wrote to the institution to express my reservations:

“…I think basing a judgement on the name or impact factor of the journal rather that the work that the scientist in question has reported is profoundly misguided. I am therefore not willing to participate in an assessment mechanism that perpetuates the corrosive effects of assessing individuals by considering what journals they have published in. I would like to be able to provide support for Dr X’s application but feel I can only do so if I can have the assurance of your head of department that the Committee will work under amended criteria and seek to evaluate the applicant’s science, rather than placing undue weight on where he has published.”

The reply was curt — they respected my decision for declining. And that was it.

I feel bad that I was unable to participate. I certainly wouldn’t want my actions to harm the career opportunities of another but could no longer bring myself to play the game. Others may feel differently. It was frustrating that the university in question did not want to talk about it.

But perhaps things are about to take a turn for the better? Today sees the publication of the San Francisco Declaration on Research Assessment, a document initiated by the American Society for Cell Biology (ASCB) and pulled together with a group of editors and publishers.

The declaration, which has already been signed by over 75 institutions and 150 senior figures in science and scientific publishing, specifically addresses the problem of evaluating the output of scientific research, highlights the mis-use of impact factors as the central problem in this process and explicitly disavows the use of impact factors. I can hardly believe it. This is the research community, in its broadest sense, taking proper responsibility for how we conduct our affairs. I sincerely hope the declaration becomes landmark document.

All signatories, whether they be funding agencies, institutions, publishers, organisations that supply metrics or individual researchers, commit themselves to avoiding the use of impact factors as a measure of the quality of published work and to finding alternative and transparent means of assessment that are fit for purpose.

The declaration has 18 recommendations — targeted at the different constituencies. The first one establishes its over-riding objective:

“Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.”

The remainder go into more detail about what each of the different players in the business of science might do to escape the deadening traction of impact factors and develop fairer and more accurate processes of assessment. By no means does this spell the end of ardent competition between scientists for resources and glory. But it might just be a step towards means of evaluation that are not — how shall I put it? — statistically illiterate.

I urge you to download this document (available as a PDF) read it and circulate it to your colleagues, your peers, your superiors and those junior to you. Tell everyone.

And of course, you should sign it.

Update 17th May, 18:28 — I have been discussing my decision — mentioned above — not to participate in the review of a promotion candidate over at Drug Monkey’s blog. He is very critical of my stance and I think may have a point (see his comment thread for details). As a result, while I have not changed my view of the reliance that the selection procedure at the institution involved places on the journal names,  I emailed them this morning to offer my services as a reviewer (their deadline has not yet passed). I also pointed out this blogpost and Drug Monkey’s reply by way of explanation but also with a view to pursuing a discussion about their selection process. If they take me up on my offer, I think I can provide a review and incorporate into it my concerns about the implicit reliance on journal impact factors.

I have a problem with Excel, or rather, with a particular spreadsheet that I would like someone to solve elegantly.

You can download a version of my spreadsheet here. The image below shows the contents. The ‘real life’ spreadsheet will be much bigger but I would like to email it to every person named as a Supervisor, Examiner1 or Examiner2. In principle, each recipient’s name could appear in any of these three columns. I’ve highlighted my own surname to illustrate this.

What I want is a text box at the top of the spreadsheet where each recipient could type their name, an action that would at a stroke reduce the spreadsheet to just those rows that contain their name. I’m guessing this could be done with a macro of some sort. I don’t know. Excel baffles me.

Care to have a try? As an incentive I will send a £10 Amazon voucher to the first person to send me a copy of the spreadsheet that has this function. I promise. You can find me at s dot curry at imperial dot ac dot uk.

OK Internet — go!

Results time — 16th May, 09:00 

Thanks to all the wonderful people who sent in solutions — what an industrious and inventive lot you are. I got solutions from fifteen different people which I wanted to summarise and share. I’m presenting them more or less in order of receipt (with links to the files), which seemed also to track the level of sophistication. At the end, I’ll announce the winner.

Once of the first entries, from Matthew Russell, hit on a solution that I had implemented crudely myself. The trick is to create an additional column that contains a flag (0/1 or yes/no) to indicate whether the name being searched for occurs in any column:

Having typed in the name you are searching for, you click on the down-arrow attached to the flag column and use the filter function that appears to filter (on ’1′ in this instance) and so reduce the display to only rows containing that name. Job done. Several others — Dorothy Bishop, Thomas Phillips, Steve Black and Pierre Clavel — came up with similar solutions.

Alan Henness produced a variant on this approach which involves generating a concatenated list of the contents of the columns being searched in a separate column which can then be filtered in the same way as above to find the name you are looking for:

In this case you click on the down-arrow attached to the Concatenated column and filter on the name you are looking for. Peter Binfield also produced a solution like this. It works well but probably would be rather tricky to implement if you had a large number of columns to search.

These solutions require the user to invoke the filter command but I was really looking for a simpler solution (not trusting to the Excel capabilities of myself or my users). Siobhan Clibbens came up with a nice button-driven implementation that relies on macros. Here, you type in the name you are searching for and click on the easy-to-spot ‘Filter’ button to show only those rows with the search string. Clicking the ‘Clear Filter’ button resets the spreadsheet to its original form.

Others sent variants of this button approach including Stephen Royle, Kevin Marshall, and @BenMMiles house mate (still nameless!) but to my eye Siobhan’s had the neatest user interface.

Stuart Cantrill sent a slightly modified version of this approach which, rather than relying on buttons, presented instructions on keystroke combinations to control the filtering of the spreadsheet.

Matthew Russell produced a second entry (keen!) that streamlines this approach even further. There are no buttons to press, you simply enter the name you are searching for and hit return. The filtering occurs automatically. To reset, you clear the text entry box and hit return again. Nice.

The most radical solution was sent in by Christian Cole who argued strongly on Twitter that Excel is not the tool to be using for this sort of data handling and sent in a web-based solution (zipped file) that relies on HTML and javascript. It works beautifully – the rows of the table (database?) collapse as you type:

I realise I am a lowly Excel neophyte so my commentary on this is almost worthless but I was very impressed by the technical skill on show here. Equally splendid was the willingness of so many people to rise to the challenge. Several commented that they weren’t interested in the £10 prize money, but had simply been driven by the challenge. That’s great news because next time I may not have to offer any reward!

But there has to be a winner and so the prize goes to Siobhan Clibbens for the combination of the particular elegance of her solution and her speed of submission (Update: see below for modifications suggested by Tom Grant that will make it scalable). Honourable mentions to everyone else.

Ultimately, which solution I implement to solve my real-life problem with depend on how easy it is to adapt to the bigger spreadsheets that I have to handle, but I feel I have made some important steps along the learning curve and that a great deal of help is readily available.

Thanks again to everyone who participated. Internet FTW, as I believe people like to say.

Update 18th May, 11:13 — Tom Grant emailed to provide a method for adapting the winning solution to make it work for any length of spreadsheet:

‘Open the VBA editor in Excel (Tools > Macro > Visual Basic Editor) and then select “Module 1″ from the Project bar on the right, the code should show as follows:

——————————

Sub Filter()
‘
‘Filter Macro
‘Advanced filter across columns B, C and D so that only rows where the name in cell B1 appears in one of these columns are shown.

‘ Keyboard Shortcut: Ctrl+f
‘
Range(“C10″).Select
Range(“A8:D21″).AdvancedFilter Action:=xlFilterInPlace, CriteriaRange:= _
Range(“A3:D6″), Unique:=False
End Sub
Sub Clear()
‘
‘Clear Macro
‘
‘ Keyboard Shortcut: Ctrl+Shift+C
‘
ActiveSheet.ShowAllData
End Sub

——————————

If you replace the line

Range(“A8:D21″).AdvancedFilter Action:=xlFilterInPlace, CriteriaRange:= _

with

Range(“A8″, Range(“A8″).End(xlToRight).End(xlDown)).AdvancedFilter Action:=xlFilterInPlace, CriteriaRange:= _

this will make it good to go for all list lengths.’

Thanks Tom!

Update 19th May, 13:45 — Some people just can’t stop themselves. Steve Black has re-worked Matthew Russell’s second solution (mentioned towards the end of the blogpost) to make it work for any size of table. If I have understood correctly, Steve’s solution is an ‘Excel Binary Workbook’ (hence the .xlsb extension on the filename) and relies of some Visual Basic code, which you can access from the menu Tools > Macro > Visual Basic Editor.

I’ve tweaked it slightly, to add a spacing row at the top and a short instruction on how to reset. Adding the spacing row necessitated a small edit of the code. My changes are in bold below – I set the ActiveSheet.Range (the area highlighted in pale yellow) to start at cell a4 and subtracted 3 from the UsedRange.Rows.Count in the same line to take account of the fact that there are now 3 rows in the spreadsheet before you get to the data that are to be analysed:

Private Sub Worksheet_Change(ByVal Target As Range)
If Not Intersect(Target, Target.Worksheet.Range(“b1″)) Is Nothing Then Call HideRows(Target)
End Sub

Sub HideRows(TargetName As Range)
Dim cell As Range, rTable As Range

‘set rTable to be active range of table
Set rTable = ActiveSheet.Range(“a4“).Resize(UsedRange.Rows.Count – 3, UsedRange.Columns.Count)

‘unhide all if no name supplied
If ActiveSheet.Range(“b1″).Text = “” Then
rTable.Rows.Hidden = False
Exit Sub
End If

‘hide all rTable.Rows.Hidden = True

‘only unhide if match found
For Each cell In rTable.Cells

‘test whether cell contains required (ucase ensures capitalisation doesn’t matter); NB also checks student row for simplicity–this might have side effects
If UCase(cell.Value) = UCase(TargetName.Value) Then
cell.EntireRow.Hidden = False
End If

Next

End Sub

Steve recommends cutting and pasting your real-life data into a copy of this spreadsheet to replace the dummy data. To ensure the above version works, you will need to have the data start in cell a4 and have only 3 rows above for your text and headers. Otherwise, you’ll need to edit the code as I have above.

The reasons why some scientists succumb to corruption have no doubt also intrigued psychologists but of late you could be forgiven for suspecting them of being more preoccupied with committing fraud than analysing it. Psychology is not the only field of inquiry tarnished by incidents of dishonesty — let’s not forget physicist Jan Hendrik Schön, stem cell researcher Hwang Woo-suk or crystallographer HM Krishna Murthy — but its practitioners may be better placed than most to analyse the origins of the problem.

Indeed one of the most prominent recent transgressors has provided some useful insights. In 2011 Diederik Stapel, a professor of social psychology, was suspended from his job at Tilburg University because of suspected fraud; a subsequent investigation found that he had fabricated data over a number of years that affected over 55 of his publications. Interviewed in the New York Times by Yudhijit Bhattacharjee, the disgraced psychologist was candid about where he had gone wrong:

Stapel did not deny that his deceit was driven by ambition. But it was more complicated than that, he told me. He insisted that he loved social psychology but had been frustrated by the messiness of experimental data, which rarely led to clear conclusions. His lifelong obsession with elegance and order, he said, led him to concoct sexy results that journals found attractive. “It was a quest for aesthetics, for beauty — instead of the truth,” he said. He described his behavior as an addiction that drove him to carry out acts of increasingly daring fraud, like a junkie seeking a bigger and better high.

There’s a fair bit to unpack in those few lines. In part the problem is systemic. Stapel’s allusion to journals’ demands for ‘sexy results’ is a nod to one of the corrosive effects on researchers of the construction of journal hierarchies on the shifting and unreliable sands of impact factors. Stapel elaborates later on in the interview:

What the public didn’t realize, he said, was that academic science, too, was becoming a business. “There are scarce resources, you need grants, you need money, there is competition,” he said. “Normal people go to the edge to get that money. Science is of course about discovery, about digging to discover the truth. But it is also communication, persuasion, marketing. I am a salesman.”

Competition for finite resources is no bad thing, helping to ensure that grants and promotions are awarded to the applicants doing the highest quality science, but the process has been undermined by over-reliance on journal impact factors as a measure of achievement. A paper in a ‘top’ journal is now often seen as a more important goal than the publication of the very best science because busy reviewers rely too readily on the name of the journals the applicants’ papers are published in rather than the work that they report. Although ‘many normal people go to the edge’, it is clear that Stapel went well beyond it. At some point the self-promoting salesman overtook the discoverer of truth.

Unfortunately the issue of publication pressures leading to poor scientific practice is hardly news. A decade ago Peter Lawrence — always worth reading on the conduct of science and scientists — analysed the ‘politics of publication‘ and lamented that “when we give the journal priority over the science, we turn ourselves into philistines in our own world.” Lawrence’s gloomy prognosis has been borne out by Fang and Casadevall’s revelation that retraction rates are strongly correlated with impact factors. Stapel’s unmasking continues that sorry trend, one that will not be reversed until we can break our dependency on statistically dubious methods of assessment.

Problems of dubious practice (of varying degrees of severity) are more widespread than most realise but It is still true that most scientists live with the stress of competition without relinquishing their ethics. So what pushed Stapel over the edge? Good mentorship of junior scientists is recognised as a valuable corrective but the Dutch researcher’s training is not discussed in detail in the New York Times interview. He himself seems to think that it was the interaction of his personality traits with the highly tensioned system of publication and reward that led to impropriety. His “lifelong obsession with elegance and order” appears to have been at the root of his frustration with “the messiness of experimental data, which rarely led to clear conclusions”.

Stapel is hardly alone in his desire for elegance. Many scientists will have felt the deep satisfaction of conceiving a theory that brings a graceful simplicity to unruly data or of executing experiment that confirms a new hypothesis. There is an almost visceral pleasure in such instances of congruence, and aggravation in equal measure when experiment and theory collide abortively. Thomas Henry Huxley identified the tragedy of science more than a century ago — “the slaying of a beautiful hypothesis by an ugly fact” — but it was for him something you simply had to live with.

However, Huxley’s aphorism belies a more complex truth because science is a messy business and it is not always clear when a fact is truly ugly enough to bring down a hypothesis. The judgement can be a fine one and observations are sometimes set aside quite properly as part of plotting an intuitive path to a new insight; but the process is clouded by the degree of conviction that the scientist has in their cherished hypothesis, so the handling of inconvenient truths can shade into malpractice.

Crick and Watson were up-front about the need to discount some of the data that they worked with en route the structure of DNA — ‘some data was bound to be misleading if not plain wrong’, wrote Watson — but others have dissembled*. Mendel, Millikan and Eddington, for example, all discarded observations that famously conflicted with their respective conclusions on heredity, the charge on the electron and the veracity of Einstein’s general theory of relativity (but see update below with regard to Eddington). As Michael Brooks has pointed out in Free Radicals, his entertaining book on rule-breaking researchers, these renowned scientists may have been vindicated by history but their shady practices were hardly justifiable at the time. Stapel’s misdemeanours of fabricating data to support his hypotheses are more extreme — he also loses out also because his theories of psychological priming have been undermined by his unmasking — but nevertheless lie on a continuum of fraudulent practice with his scientific forebears. They all share the belief that they were right.

Even so, I can’t quite get the measure of Stapel’s behaviour. Perhaps the success that flowed from his synthetic results, given the seal of approval by peer reviewers and editors when published in prestigious journals, validated an approach that he must have known was scientifically dubious. The New York Times interview conveys a sense of regret now that he has been found out — a regret sharpened by the reaction of his wife, children and parents, forced to look anew at a man they knew so well — but why did he never question himself during the years of fabrication?

In my mind I keep returning to Stapel’s dissatisfaction with the untidiness of experimental data. I think that might be because I have just published one of the messiest papers ever to come out of my lab and am rather pleased with it for precisely that reason. I offer this story as a counter-anecdote to the case of the errant psychologist, not as a holier-than-thou pose, but simply to give a sense of what it feels like to wrestle with real data.

Our paper reports the structure of a norovirus protein called VPg. Though long supposed to be ‘intrinsically disordered’, our work shows that the central portion of VPg’s chain of amino acids folds up into a compact structure consisting of two helices packed tightly against one another; the two ends of the protein remain flexible. It’s nice to confound the prevailing viewpoint on VPg but that’s not the interesting bit about our new results.

The VPg protein — a pair of nicely packed helices

The interesting bit is that our structure doesn’t make sense. Not yet at any rate. Usually, working out the structure of a protein is an enormously helpful step towards figuring out how it works but that’s not the case with VPg. Our structure is a bit baffling.

The protein plays a key role in the virus replication, the process of reprogramming infected cells to make the components — proteins and copies of the viral RNA genome — needed to assemble thousands of new virus particles. That’s what infection is all about, at least as far as the virus is concerned (though the infected host often has a different perspective).

VPg acts as seed point for the initiation of the synthesis of new viral RNA genomes. To do this it is bound by the viral polymerase, an enzyme or nanomachine that catalyses the chemical attachment of an RNA base to a specific point — a tyrosine side chain — on the surface of protein. In turn this RNA base becomes the point of attachment for the next one and so on until the whole RNA chain — all 7500 bases — is complete.

From our structure we can see that the tyrosine anchor point on VPg lies on the first helix of the core structure but the problem is that the core is too big to fit into the cavity within the polymerase where the chemistry of RNA attachment occurs. So at first sight, VPg appears to have a structure that interferes with one of its most important functions. To solve this apparent contradiction, we came up with what I thought was a rather lovely hypothesis: we guessed that the VPg structure has to unfold to interact properly with the polymerase, supposing there might be just enough room for a single helix to get into the active site but not a tightly associated pair.

VPg: too bloody big to fit in the polymerase active site!

We tested this idea by mutating our VPg to introduce amino acids changes that would destabilise its core structure, reasoning that this would make it easier for the polymerase to grab on to the protein, so increasing the rate at which it could add RNA bases. But although the changes made disrupted the protein structure, they almost invariably also reduced the efficiency of the polymerase reaction. The experiment succeeded only in generating an ugly fact to disfigure our hypothesis.

Except it’s not dead yet — not to me. I can make excuses. The method we used to measure the rate of addition of RNA to VPg by the polymerase was less than optimal. We couldn’t work with purified components in a test tube, and so had to monitor the reaction inside living cells using an indirect readout for elongation of the RNA chain. It remains possible that this assay is confounded by the effects of other molecules in the cell. Plus, we haven’t yet been able to analyse the structure of the viral polymerase with VPg bound to it — caught in the act of adding RNA bases. Like Thomas, until I can really see evidence that conflicts with my supposition, I’m not ready to give up on the hypothesis that VPg has to unfold to interact properly with the polymerase.

But it could take quite a while to develop the reagents and the techniques to do these more probing experiments and since we had already spent quite a number of years getting to this point, we wanted to publish the results. The story we had to tell in the paper in unfinished. To some eyes it might look like a bit of a mess and I was certainly concerned that the reviewers of the Journal of Virology, where we eventually submitted the manuscript for publication, might insist that we go back to the lab to get the data to fill in the gaps. We had an interesting new structure to report but our experimental analyses had only managed to confirm that we don’t yet know what the structure is for. We were asked some searching questions and the manuscript was improved by the subsequent editing but happily the reviewers — and the editor — still understood that progress in science is more often made in small steps than in giant leaps.

We haven’t tied off the whole story of how VPg in norovirus RNA replication but that’s OK. Now that we have given an honest account of our puzzling structure, others can also apply their minds to the problem. Indeed the publication has already sparked a couple of interesting email exchanges. The situation might still be messy but it’s far from messed up.

Update, May 12: As pointed out by Cormac in two comments below and by Peter Coles on twitter (see my reply below), there appear to be strong arguments for not including Eddington in this list of dissemblers. It is ironic perhaps that a blog on messiness in science should itself become rather messy but I prefer to think it merely shows the value of open discussion.

*Of course, Crick and Watson famously benefitted from not entirely proper access to Franklin’s and Gosling’s X-ray diffraction images of DNA.

Within days I should be able to remove the Libel Reform Campaign button from my web-site because late yesterday afternoon the Defamation Bill had its final reading in the House of Lords. It should pass back to the Commons today for approval (but see updates below!) and then proceed to the statute book.

Lord McNally leading the final Lords debate on the Defamation Bill

What a long and rocky road it has been since the campaign started in 2009. Even at the end, it looked as if the bill might be derailed. First, by an amendment introduced by Lord David Puttnam which added provisions on privacy in order to provoke the Government into addressing the legislative challenges to press freedom raised by the Leveson report. That move, which lacked cross-party support and seemed likely at one point to prevent the Defamation Bill being reintroduced to Parliament, was eventually resolved after long night of political horse-trading.

Then on Tuesday last week amendments approved by the Lords which sought to oblige companies to demonstrate real financial damages before suing for libel and to prevent firms contracted to provide public services from using libel law to stymie criticism (echoing a principle established for public bodies) were undone in the House of Commons by a Government-supported amendment introduced by Sir Edward Garnier MP.

It was all getting very complicated — opinions varied on whether the battle was lost or won; see, for example, the blogposts published following the Commons debate by David Allen Green and Prateek Buch.

And then on Monday came news of a government U-turn on the question of enacting a stricter financial test on companies to demonstrate real damage before they could bring a libel action. That volte-face duly transpired in the debate in the Lords yesterday where a government amendment providing for such a test was approved. Why the government changed its mind in the past week remains a mystery to me, as do many aspects of parliamentary language and procedure.

Nevertheless the upshot is that we are on the verge of having significantly reformed libel laws in England and Wales. Not everything the campaign wanted has been won but there is little doubt that the law will change for the better. With enactment of the bill we will have a stronger public interest defence, protection for peer-reviewed academic publications, a test for real financial damages to prevent libel chill and reduce the power imbalance between powerful organisations and private individuals, and a legal framework that is updated for the internet age.

I cannot pretend to understand the exact nature and implications of all the provisions of the final bill — I hope that the legal bloggers will soon weigh in with analysis of what exactly has changed (see final update below*) — but yesterday was most definitely a good day for free debate.

Having observed from near and far over the past few years, it has been an interesting and remarkable journey — a heady mix of social media, chiropractic, celebrity endorsement and courtroom drama, not to mention the unseen hours and hours of dogged campaigning. Special gratitude must go Simon Singh for having the courage to face down the libel threat from the now discredited British Chiropractic Association, a move that was key to igniting the campaign, but particularly also to the folks at Sense About Science, English PEN and Index on Censorship for gathering public support for a campaign that has changed the law. It feels like… democracy.

Update (09:51, 24-4-13): Perhaps I wrote too soon because just moments after publishing this post this morning came news of another 11th hour amendment from Sir Edward Garnier MP that aims to reverse the government-backed change made to the Defamation Bill in the Lords yesterday. It is to be hoped that the government will secure support for its own amendment in the Commons this afternoon but we shall have to wait and see…

Update (14:50, 24-4-13): …fortunately, Garnier could read the writing on the wall and, in the end, did not press for a vote on his amendment. Immediately thereafter, and just a few moments ago, the Commons voted to approve the final amendment agreed yesterday in the Lords. And so ends the Parliamentary journey of the Defamation Bill. It should soon reappear as the Defamation Act, 2013. 

Bottomley is amused to be defamed (under Parliamentary privilege) by Sir Edward Garnier

*Update (21:06, 24-4-13): The Libel Reform Campaign has broadly welcomed the new legislation while also pointing out some of the missed opportunities. For more detail, see the useful summary (PDF) of the strengths and weakness of the bill that has been passed. Some things will depend on the nature of the procedures to be introduced by government. Dr Evan Harris, who has been involved in the campaign from the start, scored the bill at 19/33 but thought that might rise to 26/33 depending on how new rules and regulations are implemented) 

The academic community tends to view peer reviewed journal articles as the most important thing to considered when evaluating a contribution, or an individual. But is this actually the best we can manage? Or can we apply modern tools and more enlightened thinking to come up with new and improved ways to measure a contribution to the scientific enterprise?

Journal articles are regarded as the ‘minutes of science’ - a supposedly perfect (and permanent) formulation of ‘the final answer’. And once an article is published, it is typically measured by a simple count of the number of citations it receives (or, disturbingly, to measure its worth based on a count of the citations that other articles in the same journal happened to receive).

And yet, the journal article is only a single point in time in the lifecycle of a piece of work. When we only judge an article, we ignore the individuals behind it. And when we judge individuals based only on the article they have written, then we do not take account of the processes (both good and bad) that have led up to that point, or beyond. And when we judge those articles based only on scholarly citations, then we do not take account of the myriad of other ways that the publication has contributed to the scientific enterprise. In all of these ways, we believe that the process can be improved thanks to the development of new tools and new ways of thinking.

The typical lifecycle of any new finding is to research it; to discuss and develop it; to formally publish it; to influence others by the act of publication; to accrue recognition for having done the work; and to learn from the experience when starting the next research project. Let’s deconstruct each step.

The research: Any research article starts with original research, and typically this research is kept private – confidential to the researcher and their lab. And yet, there is an increasing belief that if you shine a light upon a process, the process itself is improved. As a result, there has been a slow, but growing movement towards ‘open notebook science’, which is an attempt to persuade researchers that by openly sharing their original research, in real time, they can improve their work. Although this degree of transparency may be too much for most researchers, it is a movement which is slowly gaining traction, and it is clearly an attempt to document the origin and development of a scientific thought at the earliest possible stage.

Discussion and Development: Once the research is conducted, the next logical stage in the process of documenting a piece of work is to develop a draft, often in the form of a preprint (or, in some fields, an abstract or a poster at a conference) and to develop that draft in light of feedback. A preprint is simply an early version of something which will later evolve into a formal publication and is typically un-peer reviewed. Many fields have retained a preprint culture (for example the physics, mathematics, social sciences and economics fields all have active preprint servers), however the biological and medical fields are notable for their absence. But if a preprint culture were to flourish in the biological and medical sciences, it could potentially be the missing link in the story that this post explores.

In a recent Scientific American guest blog post, we explored some of the reasons for the lack of a preprint culture in the biological and medical sciences. In that post we explained that we at PeerJ have put our money where our mouth is, by launching a new preprint server for the biological and medical sciences (“PeerJ PrePrints”) and why we think this can now be successful. If we can’t persuade the biological and medical fields to adopt a culture of open notebook science, then perhaps we can at least persuade them to adopt a preprint culture. Certainly these disciplines have been at the forefront of the Open Access movement, and so it is not too great a conceptual leap to go from openly sharing a final published article, to openly sharing the earlier drafts that led up to it.

The Formal Publication – Not the End of the Story: We mentioned that the journal article of today is a fossilized manifestation of the ‘minutes of science’. If we accept that we can show the evolution of the finding, through discovery (in an open notebook) and early formulations (in a preprint), then why should we accept that the evolution of this work stops at the moment it is published in a journal. Surely a journal article can be in error and require revision, or new results might strengthen or weaken the case? Instead of publishing an entirely new article (simply to put another ‘counted’ publication into your resume) why not revise and extend the existing publication? Again, this might be asking for too much from today’s scholarly society, but once again we can see that the ability to publish drafts (aka preprints) and to measure their unique contributions, could be applied to this use case as well.

Measuring the Contribution: Many people are now building what they hope will be better tools to evaluate academic contributions. Most visibly, this manifests itself in the ‘altmetrics’ movement (also known as ‘article level metrics’ when only concerning itself with an article). Altmetrics attempt to measure every way a piece of work might influence the wider world – be that scholarly citations; mentions in Wikipedia; media coverage; tweets; online usage; blog posts; or changing government policy. This approach has great promise as a way past the impasse caused by simply counting one metric (the scholarly citation) and is being increasingly used by publishers such as PLOS, BMC, PeerJ, Frontiers and eLife to provide richer metrics on their published articles. But the potential is far greater than this limited scope might show – the beauty of the approach is that it can be applied to any digital object - it can be applied to open notebooks, to data sets, to software code, and to preprints too (all of which typically receive zero scholarly citations).

Gaining Feedback, Discussing, Debating and Learning: It is important to talk about the ‘discussion’ that goes on around a piece of work, or the questions and answers that are posed to experts in the field. A finding could be the most impressive one in history, but if an author refuses to respond to criticisms, or does not engage with their follow academics in developing the work further, then an opportunity has been lost.

One solution to this problem is to open up the peer review itself process to the broader public in a practice known as ‘open peer review’. With open peer review, we can encourage (or require) reviewers to provide their names, and publish the reviews alongside the published article. This opens up a previously secretive process to maximum transparency, it means that reviewer contributions are not lost to posterity, and it means that reviewers can get credit for having performed their review. At the same time, authors can show the evolution of their work and how it has been improved by appropriate peer review. At PeerJ we operate a form of Open Peer Review (along with several other publishers such as BMC, eLife, the BMJ etc) and the reception has been overwhelmingly positive.

Another way to track the discussion, of course, is with altmetrics. Even if the publisher doesn’t provide their own ability to comment on an article, people will comment in spaces of their choosing – on twitter, facebook, or in a blog post. With altmetrics, those conversations too can be collated and evaluated.

Where Does This Leave Us: Imagine for a moment, the tenure or promotion committee of the future. When evaluating a candidate, then instead of scanning down a list of where (or alongside whom) they have published, the committee insteads scans down a list of their actual contributions and specific findings (things which today are only available in formal publications). They can see which findings were well received by the community; which generated the most debate; which ones were most rigorously defended by the author. They can drill backwards in time to see where the idea originally came from; they can see whether or not an author is openly sharing their work; putting it into the world for early feedback; trying to better their work through well informed revisions. They can look at a suite of metrics to see if the work was picked up and reused by others; did it inform governmental policy; was it read and commented upon by top academics; did it have a true impact? And from start to finish they can see and track the entire lifecycle of each contribution – from original idea, through early drafts; formal publications; and subsequent revisions and iterations.

This future scenario may actually be the nightmare scenario for the people who serve on those committees, and certainly it is a lot more complex than the situation as practiced today. But if it can be done correctly, then surely it would provide a more holistic and nuanced evaluation of the contribution of an individual, or of their work. Evaluation ‘in the round’ not ‘at a point’.

Researchers would no longer be thought of as simply ‘authors’ (which is only the end point in their process of research) – instead they can demonstrate their entire contribution and position within their community. They can show how their data sets or software commits contributed, for example. They can show how they are a good actor within the community – commenting on, and helping to improve the work of others. They can finally get credit for work that previously went unnoticed in today’s environment which only cares about their published output.

At PeerJ, we consciously avoid a measurement of significance, impact, novelty, or degree of advance before publication. Instead we are building tools which will allow the community to determine these things for themselves by looking at the work ‘in the wild’ (and increasingly ‘in the raw’). With our new preprint server (PeerJ PrePrints) people are able to ‘show their workings’ and place an early draft in front of their community for broader feedback (using feedback functionality which has just been launched); the peer review process of the PeerJ journal then asks reviewers to only comment on the validity, or soundness, of the science presented and at the same time promotes transparency by encouraging reviewers to name themselves; authors are encouraged to publish their peer review and revision history alongside their article; and finally the published article (as well as any preprint versions) is made available with a suite of article level metrics including usage data, referral stats, social media mentions, scholarly citations and so on. And throughout every interaction with the PeerJ system, people can accrue credit for their activities as authors, reviewers, editors, or commenters using a new system of ‘user contribution’ points. As can be seen from the arguments above, we believe that by encouraging this level of transparency, and by exposing the lifecycle of a scholarly manuscript, the work of academics can be more fairly evaluated and more effectively built upon.

Of course, we are fully aware that change like this will not happen overnight, but we do believe that change of this nature is now underway. We see the growing popularity of preprint servers like the arXiv, SSRN, or RePEc (and our own PeerJ PrePrints); we see the development of new sharing tools such as Mendeley and FigShare; we see a string of experiments in the peer review space such as PeerJ, Rubriq,  F1000 Research and eLife; we see successful companies being launched in the altmetrics space such as Impact Story, Altmetric, and Plum Analytics; and we see the now unstoppable rise of the open access movement which will place all content online in a maximally sharable format. It feels to us like all the pieces are now in place. It simply needs the will of the academic community to embrace the change, and to move themselves forwards.

“When assessing proposals for research funding RCUK considers that it is the quality of the research proposed, and not where an author has or is intending to publish, that is of paramount importance” 

RCUK’s Alexandra Saxon was good enough to make particular mention of our request in her blogpost to explain the most significant revisions in the new guidelines. I’m also grateful to Peter Coles (aka @telescoper) for noticing.

This is only a small step on the road to elimination of the pernicious effects of impact factors on our processes of assessment. There is no case for resting on laurels. We still need leading scientists, other funders, universities and journals to listen to the mood music and respond in a like manner and suggestions for how to achieve that will be most welcome.

As we now know, Alexandra left a comment on that post indicating that RCUK will respond positively by amending the guidelines in accordance with our request. I am grateful to RCUK for such swift action on this and look forward to the revised text with great interest.

Dear Alexandra

I am grateful for the opportunity to comment on the recently published guidelines on the new OA policy of RCUK.

I appreciate the flexibility built into the guidelines, which usefully reiterate the principle that the choice of route to OA remains with the authors and their institution. As the document recognises, this freedom of choice should be an important factor in helping to drive down costs in the developing market for OA publishing, something that is in the long-term interests of both researchers and funders.

For such a move to succeed, the guidelines ask researchers to exercise some adaptability in their choice of journal. However, this is only likely to happen if the community can free itself from its dependency on impact factors. Although, as far as I know, neither RCUK nor any of its associated research councils has an explicit policy of considering impact factors in the assessment of funding applications, the practice of using them to quickly evaluate individuals is widespread and unjustified.

Therefore, to help make a shift away from the culture of dependency on impact factors, I would like to ask RCUK to amend its guidelines to include a statement confirming that Research Councils will explicitly disavow the use of impact factors in the assessment of individuals or applications.

As you know I raised this issue in a recent blogpost, where the rationale behind this request is laid out in more detail. I also used this blogpost to recruit support for my request and all those who indicated their agreement, either in the comment thread or directly by email, are listed as co-signatorties here. That list would no doubt have been longer if you hadn’t offered prompt and welcome reassurance that action to comply with this request would be taken. We look forward to the revised document.

Best wishes,

Stephen Curry, Imperial College
Tom Olijhoek, OKF Open Access Working Group
Jon Butterworth, UCL
Richard Harvey, Kings College London
Fional Nielsen, Illumina UK
Peter Quinn, Kings College London
Dave Barlow, Kings College London
Richard Johnson, Swansea University
Sylvia McLain, University of Oxford
Athene Donald, University of Cambridge
Dorothy Bishop, University of Oxford
M Jayne Lawrence, Kings College London
Tamsin Mather, University of Oxford
Peter Murray-Rust, University of Cambridge
Graham Steel, Glasgow
Mike Taylor, University of Bristol
Steve Pettifer, University of Manchester
Paula Salgado, University of Newcastle
Chris Chambers, Cardiff University
Julia Bardos, University of Cambridge
Ross Mounce, University of Bath, OKF
Mike Fowler, Swansea University
Alfonso Martinez Arias, University of Cambridge
Samuel Furse, University of Nottingham
Elizabeth Stanley

From overseas:
Jim Till, Ontario Cancer Institute, Canada
Jonathan Peelle, Washington University, USA
Jim Woodgett, Samuel Lunenfeld Research Institute, Toronto, Canada

 

15-3-2013: Please see update at the foot of this post for an important announcement from RCUK.

I had no idea when I clicked ‘publish’ last August that my ‘Sick of Impact Factors’ post would unleash such a huge response. Evidently I had pulled on a chain that everyone feels bound by. The post attracted over 180 comments and tens of thousands of page views. It is still getting over 2000 hits a month.

As I wrote in that post (and elsewhere), the abuse of journal impact factors (IFs) in the assessment of scientists applying for jobs, promotion or funding is a deep-seated and largely self-inflicted problem. It is retarding the uptake of open access, because the addictive lure of IFs inhibits some authors from choosing new OA journals and allows ‘high-impact’ journals to lever higher article processing charges (APC) from those paying for gold OA.

The response to the blogpost has been very gratifying but will ultimately be worthless if it cannot be harnessed to make the necessary shift away from a culture of dependency on impact factors. It seems to have influenced the thinking of at least one journal. Nature Materials cited my post in an editorial last month that warned its readers of the dangers of using IFs as a guide to the performance of individual researchers. I was pleased also to see that the journal’s Instructions to Authors now links to this editorial.

That’s a start and I very much hope other journals will follow this fine example. But for the shift to take hold we also need influential players such as universities and funding agencies to publicly disavow the use of impact factors in the assessment of individuals.

We should not underestimate how long this might take. In guidelines for the Research Excellence Framework (REF), which is currently gathering information on outputs to assess the quality of UK research, HEFCE has stated clearly that journal impact factors will not be used by its judging panels. However, there remains widespread distrust in the research community that this will actually happen. An informal survey of departmental practices around the country by Dr Jenny Rohn found that many are looking at IFs when deciding which of their researchers’ publications to submit to the REF. Clearly, old habits die hard.

But now there is a new opportunity to hammer one more nail into the impact factor coffin. On 6th March RCUK, the body that oversees the UK research councils, issued updated guidance on its new open access policy, which is due to take effect on 1st April this year. This policy has been much debated but I don’t want to rehearse those arguments again today. Instead I want to focus on a key point relating to impact factors.

While their OA policy indicates RCUK’s preference for immediate access funded by APC payments (gold OA), RCUK-funded authors can alternatively meet their obligations by depositing their final peer-reviewed manuscript in an institutional repository (green OA). The guidelines (PDF) seek to clarify the flexibility that is available to researchers in deciding which route to follow, which may well affect the particular journals that they should choose and that inevitably raises the question of impact factors.

The guidelines make it abundantly clear that the “choice of route to Open Access remains with the author and their research organisation” but how that plays out in terms of journal choices on the ground remains tricky. Section 3.5(ii) discusses the payment of APCs from the block grants that RCUK will provide to institutions. RCUK hasn’t specified upper or lower limits on APCs that are allowable although they are keen to drive down costs*:

“institutions should work with their authors to ensure that a proper market in APCs develops, with price becoming one of the factors that is taken into consideration when deciding where to publish.”

In pursuance of this admirable goal the document notes that:

“HEFCE’s policy on the REF, which puts no weight on the impact value of journals in which papers are published, should be helpful in this respect, in that it facilitates greater choice.”

However, RCUK fails to follow HEFCE’s lead with a statement of its own.

Later in the document (section 3.6(iii)) the issue of journal choice raised again (with my emphases in bold):

“Where an author’s preference is ‘pay-to-publish’ and their first choice of journal offers this option, but there are insufficient funds to pay for the APC, in order to meet the spirit of the RCUK policy, the Councils prefer the author to seek an alternative journal with an affordable ‘pay-to-publish’ option or with an option with embargo periods of six or twelve months.”

Admirable flexibility but the guidance again fails to offer researchers explicit reassurance on the question of impact factors, which cannot at present be disentangled from the decision about which journal to select.

The remedy for this is straight-forward: the guidelines should be amended to include an explicit and public reassurance to researchers that RCUK and their associated funding councils will put in place instructions for reviewers and panel members to disregard impact factors in assessing all funding applications. Given RCUK’s evident approval of HEFCE’s IF-blind policy, I expect them to be ready to embrace an opportunity to foster a real improvement in our culture of assessment. The Wellcome Trust already has a statement to this effect built in to its open access policy, affirming the principle that:

“it is the intrinsic merit of the work, and not the title of the journal in which an author’s work is published, that should be considered in making funding decisions.”

Every little helps, so perhaps you can help me to persuade RCUK to adopt a similar statement? Together we can provide a friendly shove in the right direction.

The consultation on the guidelines is open until next Wednesday, 20th March. I will be writing to RCUK’s Alexandra Saxon on that date to request that an explicit disavowal of the use of impact factors in the assessment of researchers is included in the revised guidelines (providing a link to this post to explain the reasoning). Please feel free to write in the same vein or, if it is easier, leave a comment here stating that you are happy to be included as a signatory on my email. Or send me an email (s dot curry at imperial dot ac dot uk). Please give your name, title and affiliation. I imagine RCUK will be more attendant to the views of UK-based researchers but there would be no harm in giving a sense of the global reach of the problem of impact factors.

Update (15-3-2013; 11:51): I am grateful to Alexandra Saxon, RCUK Head of Communications, who has this morning added a comment confirming that “RCUK will add a statement similar to the Wellcome Trust’s to the next revision of the guidance, due to be published towards the end of the month.”

I had suspected that RCUK would be sympathetic to our request but it is nevertheless great news to hear of this commitment. Readers should still fee free to indicate their support; I have told Alexandra that I will still write next Wednesday to communicate our collective desire to see abandonment of the use of IFs in assessing applications. My thanks to all who have offered support so far, in comments and emails.

*I realise the RCUK’s preference for gold over green OA entails higher transitional costs in the short term but would like to set that debate aside just for today.

The meeting, which aimed to examine “the background to the new policy announced by David Willetts in July 2012, including the recommendations of the Finch working group, and (to) address the practical challenges of implementation”, attracted a large audience of research administrators, librarians, publishers, scientists and representatives of research funders to hear a good mix of speakers (PDF). I gather it had been arranged as a sister to a similar meeting organised by the Academy of Social Scientists last December.

I don’t have time to give a full synopsis of the proceedings (slides should be available of the RS website soon) but wanted to touch on the points that resonated most strongly with me.

First off, David Sweeney announced that HEFCE had launched a consultation on the role of open access for REF assessments after 2014. Though consultative, this document is by no means a blank slate. Rather, it sets out clear proposals, re-enforcing earlier statements, that the only submissions eligible for the post-2014 REF should be open access. In an interesting contrast to RCUK, HEFCE is agnostic about whether papers are published by the green or gold OA routes; Sweeney said it would be inappropriate for the organisation to give a steer to researchers on that particular point. The consultation is primarily asking for advice on various aspects of the implementation of HEFCE’s policy. As such it might seem a rather technical process but is nevertheless a further important signal that the momentum for open access keeps rolling on.

In the afternoon the meeting was visited briefly by science minister David Willetts; he made a short speech, explaining again the value he sees in a gold OA policy — immediate access and re-use rights via the CC-BY licence under a system that is transparent about the real costs of publishing — and declaring that green OA is “not a policy”. His argument is that pursuance of green OA leads to an unstable situation in which the cancellation of subscriptions (because readers have free access) drains the system of the funds needed to manage peer review and other publishing costs.

However, Willetts conceded that he had made little progress in persuading Máire Geoghegan-Quinn, European Commissioner for Research, Innovation and Science, to get the EU to adopt a gold-favouring OA policy. In the light of the US decision on Friday to ask federal agencies with R&D budget of more than $100m to prepare green OA policies, it seemed to me that the UK was looking isolated and I wrote as much on Monday. In the Q&A I therefore asked Willetts, if its advantages were so evident, why others were not jumping on the gold OA bandwagon set in motion by the UK and what he could do to promote the international coordination that will be needed to bring about global open access.

He didn’t really give an answer to the second part of my query but it was clear that he will be sticking to his gold OA guns for now, whatever may have happened in the US. What I found particularly interesting is that Willetts could not really articulate a convincing local economic case for the UK forging ahead with its gold-favouring OA policy is a world that looks increasingly green. There was some mention that the fact of having to deal with article processing charges (APCs) needed to cover the cost of gold OA might give the UK-based publishing industry a useful lead in a world that should eventually see a shift from a subscription-based scholarly publishing to one that is funded by APCs; but this argument didn’t have the feel of a primary policy driver.

So it looks as if the UK is taking an altruistic stance on this issue, which an unusual thing to find in a government policy. If I have understood him correctly, as Willetts sees things, going for gold OA now in spite of the additional costs in the transition is the right thing to do because it recognises that we will all have to make the shift to paying for publishing for APCs at some point. By running the experiment first, I think he is arguing, the UK aims to address and resolve the technical issues that will inevitably arise and hopes to learn lessons that can be shared with the rest of the world and so facilitate the transition to gold OA.

If these are his motives, the plan is indeed a bold one – and I have a sneaking admiration for its idealism. It is also risky and has already raised protests from various quarters — first, that the costs are too high for a science budget that is already extremely constrained and second, that green OA is the cheapest route through the transitional period. The former are real concerns, particularly in these austere times — but Willetts is evidently a gambling man. The latter argument also has some economic teeth but I would like to hear more from advocates of a transition based only on green OA mandates on exactly how the ultimate switch to gold OA can be made from the melee of subscription cancellations that they reckon will be the inevitable consequence of the success of their approach, particularly since green OA depends on compliance from the companies and learned societies that will suffer short-term financial losses.

The transition problem, whatever the route plotted through it, remains a tough nut to crack. No-one I spoke to at Monday’s meeting had a clear idea of how it would occur. We are on an experimental journey feeling our way more or less blindly — a source of occasional but considerable frustration. On the up side — or did I imagine it? — there was at least some sense that we’re all in this together.

Or there was until Tom Welton, Head of the Chemistry Department at Imperial College, got up to speak. His talk was full of charm and wit and light relief at the end of a long day, but nonetheless gave a insightful and wholly sobering account of the resistance towards OA among the majority of academics.

He told a rather shocking tale a student who, having been hired to do all the donkey-work of helping Imperial’s chemists to put their manuscripts in the College OA repository, met with widespread non-cooperation, resistance and even some outright hostility. The reasons for this are difficult to fathom — I didn’t quite buy Tom’s suggestion that his chemists were overly concerned about minor textual differences between their peer-reviewed manuscript and the journal version — but they are an important reminder that if we want researchers to adopt OA, we need to provide the right incentives. We need to make it feel worthwhile.

I believe that is an argument that can be won but it’s an argument for another day (or the comment thread) — this post has gone on long enough.

In the morning the Science and Technology Committee of the House of Lords (the unelected second chamber in the UK parliament) published the report of its inquiry to the implementation of a new open access policy by Research Councils UK (RCUK) in the wake of the Finch Report. The committee had taken written (PDF) and oral evidence from a wide variety of interested parties, including Janet Finch herself and David Willetts, the minister overseeing the policy.

The report is strongly critical, and is trailed as such on the Committee’s web-site. Their lordships particularly decry the confusion surrounding the implementation of the RCUK policy.

There is some justice in this, though a cynic might be tempted to remind the committee that the association of open access and confusion is nothing new. However, I think the criticism overlooks some of the attempts that RCUK has already made to communicate its new policy. I confess I haven’t yet had time to read the report in full but would like to offer some brief commentary (italicised) on the key points in the summary which is reproduced below:

The growth of open access publishing—specifically, making peer-reviewed journal articles available online at no cost to readers—is revolutionising communication of the results of research. The Government commissioned an independent working group to consider how to expand access to publicly-funded research (the Finch Group) and Research Councils UK (RCUK) revised its policy on open access following the report of this group. The revised policy has caused considerable concern in both the publishing and academic communities. Publishers are worried about specific requirements of the policy. Learned societies fear they will lose a valuable income stream which they use to support their respective academic communities. Academics are concerned about the policy taking a “one size fits all” approach, and possible unintended consequences such as lessening the quality of peer review, restricting ability to collaborate and limiting freedom to publish in the best journals. Both communities have expressed frustration that they were not adequately consulted about the policy.

This preamble does not adequately express the spectrum of opinion that exists, particularly within some quarters of the academic community (and open access publishers) who welcomed the RCUK policy and the disruptive challenge that it placed in front of the status quo.

In the light of these concerns, we conducted a short inquiry to consider the plans for implementation of RCUK’s open access policy, with a view to offering recommendations to inform RCUK’s revision of its policy guidance. We have concluded that:

RCUK must clarify its policy guidance to reflect its incremental approach to compliance in the initial five-year implementation phase of its open access policy;

Perhaps further work is required but RCUK did clarify its policy in two blogposts published in September 2012; in November that year it announced the details of how the policy would be rolled out incrementally over the next five years.

RCUK must monitor the effects of its open access policy and its Autumn 2014 review of the policy should consider:

The RCUK had already committed itself to a review of the new policy in that time-frame, publicly recognising the new policy as a ‘journey’ — in effect a kind of experiment. Again, it seems rather odd that their lordships have overlooked this. Nevertheless they have at least provided some useful points of focus for the review.

(1) whether different disciplines require different embargo periods, licences and primary models of publication, particularly in the light of evidence gathered about readership and citation half-lives;

Some account of this had already been taken since the original formulation allowed researchers in the humanities and social sciences a longer 12-month embargo period before authors’ versions of published papers could be made available in green OA repositories. Clearly some in those disciplines favour longer embargoes (and in the wake of the inquiry, RCUK announced that they would be relaxed) — but we really should be working to reduce rather than extend the delays before research is made publicly available. 

(2) whether the UK, in stating a preference for gold open access, is moving in the same direction as other countries which are mandating open access (but not necessarily gold open access);

This is a particularly key point and one seems to me to be the significant outstanding difficulty for the UK (particularly in the light of the announcement later on Friday from the US – see below). This was also a question that, in my written submission (both to the Lords and the upcoming Commons inquiry), I wanted put to David Willetts.

(3) whether article processing charges have adversely affected the number of international articles published in UK journals;

Frankly I don’t see this as a significant risk, at least as long as many international journals (Nature and Science among them) permit authors to comply with the RCUK policy by the green OA route.

(4) effects on the quality of peer review;

Again — I don’t see where this comes from. Predatory journals aside (where no self-respecting researcher would submit their work), there is no evidence to suggest that peer review is likely to suffer as open access is rolled out. Even PLOS ONE which does not consider the ‘impact’ of submitted manuscripts, has confounded critics with the average quality of its output. The most recent entrant to the OA publishing market, PeerJ, looks likely to do the same. 

(5) impact on the number of collaborations by UK researchers; and

My experience is that scientists will collaborate with whomever they need to in order to get the job done. Considerations of the technicalities of publishing do not figure at the outset of new collaborations.

(6) effects on learned societies.

This is a fair point and remains a difficult issue. But I would also like to have seen the House of Lords ask learned societies to consider how their publishing policies are helping to make publicly funded work accessible.

The Government should conduct a full cost-benefit analysis of the policy, in view of their stated preference for gold open access; and

This seems reasonable — though I wonder does the House of Lords have a record of consistently making this demand from government departments? However, it overlooks the cost-benefit analysis in the Finch report itself and the work of Houghton and Swan, both on the ultimate savings that are likely to be realised from a global switch to gold OA and the costs associated with the adoption of different OA policies during the transition from toll access to one access (green would be cheaper than gold). I would suggest the information is there to make an informed decision.

The Government should review the effectiveness of RCUK’s consultation regarding this significant change in policy.

Again, perhaps a fair point. There was in fact a consultation process on the new RCUK policy back in the Spring of 2012 when it issued a draft document for comment. But the consultation was not widely advertised as far as I can determine. I only heard about it myself by maintaining close contact with certain grapevines. The rather negative response from some humanities scholars and social scientists, fearful of what they see is a policy moulded to suit the needs of scientists (who have different funding structures and timescales), suggests that more could be done to adapt the policy — and to convince them of the longer-term value of moving to OA publishing for publicly-funded work.

The Finch Group report emphasised the need for a smooth transition to open access to avoid damaging the “complex ecology” of research communication. We echo this call. The Government and RCUK must take immediate action to address specific concerns about RCUK’s open access policy and maintain a watching brief in case mid-course corrections are required.

Well, no-one wants an unsmooth transition but it seems to me as if we are already in the middle of one. The precise mechanism of transition from where we are now to a global system of open access scholarly publishing has yet to be mapped out and remains a point of debate, even among OA advocates.

RCUK has already responded to the committee’s report, appearing to have swallowed all the bitter medicine handed out. They are due to publish revised guidance on their OA policy ‘shortly’.

But before they have a chance to do so they will surely have to absorb the announcement Friday in the US, of a White House directive that effectively extends the green OA mandate currently operated by the National Institutes of Health to all federal agencies “with over $100 million in annual conduct of research and development expenditures”.

The new US policy was heralded as a response to the open access petition that was launched back in May 2012 (which I signed). It is a significant boost to open access advocates everywhere but, as ever with this issue, things are never entirely straight-forward. The directive requires federal agencies to produce plans to enable public access to published papers (and data) but enshrines for a 12-month embargo (RCUK’s is 6 for scientific research) and has provision for agencies to extend the embargo if they can offer justification. It is also clear the money for implementation has to be found within existing budgets, though this should produce fewer financial strains than RCUK’s gold-preferring OA policy.

The US policy shift has been given a broad welcome in many quarters. Peter Suber declared ‘This is big’, and provided a brief digest of the directive, along with analysis of how the policy will interact with the Fair Access to Science and Technology Research Act (FASTR), a proposal for new open access legislation that was recently introduced into Congress and the Senate (with tighter embargo proposals than the White House directive).

PLOS also welcomed the new policy, though one of its founders, HHMI investigator Michael Eisen, who recognised the significance of the announcement but remained critical, being particularly concerned by the concessions made to publishers. As he noted, the directive has already attracted the support of the Association of American Publishers, the same organisation that dismissed FASTR as ‘boondoggle’.

Eisen’s concerns are real enough but although the world is not moving at the speed he would wish, the news from the US on Friday is tremendously important. In particular, it makes clear that there is no prospect of the US emulating the UK in the adoption of a policy for the transition period that favours gold open access. The US is clearly plotting a green route to OA that follows a road taken by most other countries (see Richard Poynder’s blog for perceptive analysis of the current international scene) and, for the first time, the UK’s gold-friendly policy is looking vulnerable.

From the beginning, Willetts has appeared to understand the need for concerted international action on open access. He recognised as much in his speech to the publishers’ association back in May 2012 when he said:

“We share common objectives with the Commission and want to ensure that a sustainable strategy is developed for Europe as a whole. I will also be discussing the whole issue with colleagues beyond the EU. Fortunately there is already a lively debate on these issues in the US, and we hope they will be implementing similar initiatives.”

Those hopes now appear to have been dashed and it looks as if a re-think is in order.

Open access retains is ability to bewilder and surprise us. This issue is by no means over and the tectonic shifts of last Friday have made things even more complex for the UK. I look forward to hearing what Mr Willetts has to say about the news from across the Atlantic at the conference on ‘Open access in the UK and what it means for scientific research’ at the Royal Society today (program).

I beg to differ. As did Gail Cardew, the Director of Science and Communication at the RI, who wrote to Nature this week to provide some data on the RI’s activities that had been overlooked in the editorial.

Unlike the editorial, the rebuttal has been kept behind a paywall, which seems inappropriately asymmetric in a debate about an institution that is an important part of our scientific heritage. To further that debate I am posting the text of Dr Cardew’s letter below. I think it would be helpful if Nature made it freely available.

“You mischaracterize the impact and continued relevance of the Royal Institution of Great Britain (RI) by presenting an incomplete picture.

In 2012 the RI delivered 87 evening events. Of the 46 held in the Faraday Theatre, the mean attendance was 288, much higher than might be expected from a small marketing budget. The thriving schools programme featured 136 lectures and workshops, reaching nearly 13,000 students last year alone. The RI runs mathematics and engineering masterclasses for schoolchildren at more than 140 UK locations. Our activities score very highly using the industry-standard Generic Learning Outcomes, which gauge enjoyment, inspiration, knowledge and understanding.

Thanks to its unique position and unrivalled heritage, the RI attracts the best scientists and science communicators across its programmes, including psychologist Stephen Pinker and physicist Brian Cox.

Even if one thinks that public talks are irrelevant in this age of “the Internet and mass media”, then the RI is still a powerful player. Our televised Christmas Lectures had an audience of 4.2 million in 2011.

The RI Channel website launched just over a year ago and showcases some 300 videos, which have so far attracted almost 1 million views. Some highlight recent RI events, others feature re-digitized footage from our archive, and there are high-quality videos from scientific institutions across the world.

I accept that mistakes made by the RI have led to the current situation. The growing popularity of its programmes — live, broadcast and online — isn’t one of them.”

 

I made a submission to the House of Lords committee — outlining what I thought were several key points. Many others did the same. So many in fact that the compiled submissions ran to 320 pages.  To spare the wits of our beleaguered MPs I have therefore prepared a much briefer submission for the House of Commons committee, focusing on just one question. It is laid out below.

SELECT COMMITTEE ON BUSINESS, INNOVATION AND SKILLS

Response to the 
Call for evidence on Open Access

Executive Summary: The committee should ask Mr Willetts, minister for universities and science, what progress has been made in convincing other research-active nations to adopt gold-friendly open access (OA) policies that align with the current UK position. If the minister is unable to convince international opinion, a rethink of UK policy may be needed to lubricate the transition to a workable worldwide system of free access to publicly-funded scholarly publications.

1. The submitter: My name is Professor Stephen Curry. I work at Imperial College London, but am writing in a personal capacity. I have been an active research scientist for around 25 years and published over 80 peer-reviewed articles. I have written extensively on open access (OA) from the perspective of a working academic.
2. The submission (paragraphs 2-8): Open access to publicly funded research is both a huge opportunity and a contentious problem. The committee will no doubt receive submissions from across the spectrum of opinion on recent policy developments in the UK. To keep thing brief I will confine my remarks to what I think is the most important of the topics identified in the call for evidence — “The level of ‘gold’ open access uptake in the rest of the world versus the UK, and the ability of UK higher education institutions to remain competitive.”
3. Following Finch, the UK Government and RCUK have arguably made a bold move in declaring a preference for gold OA. In the long run adoption of gold OA is likely to offer a major improvement in terms of cost and access over the current mixed model, which is based on journal subscriptions and partial open access  — mostly from green OA repositories.
4. However, this improvement will only be realised when there is a worldwide move in academic publishing to a model that is predominantly or wholly OA. Ultimately, journal subscriptions could be abandoned and the money used instead to pay publishers’ Article Processing Charges (APCs) required under gold OA.
5. This transformation — and we should not underestimate how radical or important it will be —is critically dependent on international cooperation, but on this key point there appears to be much confusion. None of the major research-active nations have followed the UK lead. They appear instead to be opting for green OA mandates.
6. As a result the UK policy, which is in some ways an exemplary and usefully disruptive gamble, looks increasingly out of step with the rest of the world. To my mind there is an urgent need to coordinate OA policy internationally.
7. Mr Willetts, the minister for universities and science, has talked about consulting with his counterparts in other countries. I think the committee should ask for a detailed report on what progress he has made. If the minister is unable to convince international opinion, a rethink of UK policy may be in order.
8. Even then, if there is a swing to support for green OA, an international route needs to be mapped out that clearly explains how one gets from green OA, which depends on maintenance of an already expensive subscription model, to a system of worldwide gold OA that offers the benefits of a more transparent and more efficient publishing market, immediate (unembargoed) access to the version of record and CC-BY licences to facilitate text-mining and re-use.

In three separate sessions, which you can watch on Parliament TV if you have the time, they quizzed representatives of universities, of funding agencies and David WIlletts, the minister for universities and science, on the Finch Report and the roll-out of the new RCUK policy which has set a new course for open access in the UK.

I was only able to catch brief snatches of the proceedings and so don’t have a global view. From what I did see, it was slightly disappointing to witness so much confusion on basic points. No-one seemed to be quite sure whether the journal Science is compliant with the new RCUK policy; it is. Rick Rylance, Chair of RCUK, even suggested to the committee that the RCUK open access policy document issued last summer following publication of the Finch Report was a draft (but it wasn’t – and a clarification had to be issued later in the day).

Those errors aside I was disappointed also not to see more enthusiasm for the ground-breaking stance that the UK has adopted. Leaving aside for a moment the ongoing arguments about the detail (which I don’t wish to discount), there was little recognition of the fact that the UK has drawn a lot of attention to the issue of open access in its recent policy developments and that part of the aim of that process is to help catalyse a global transformation in scholarly publishing.

Perhaps it is the primary duty of the committee to be concerned about the impact of the policy on UK research but there was a troubling preoccupation about global university rankings and possible limitations on UK researchers’ freedom to publish in ‘top journals’. This is a further demonstration of the corrosive nature of the journal impact factor on our research culture. At no point did I hear any declarations that is it the quality of the science itself that really matters. And so the deliberations of the committee came across to me as unnecessarily parochial. It fell to Willetts in the last session to try to talk up open access as a grand project, though even he was not wholly convincing on how the delated UK preference for gold OA is going to catalyse policy transformations in the rest of the world where there is a greater inclination to green.

I was left with the sorry realisation that there remains a great deal to do to make global open access a reality.

My spirits were revived somewhat by two pieces of news from outside the committee room. The first  was about the establishment of an Open Library of Humanities, which aims “to provide an ethically sound and sustainable open access model for humanities research”. After weekend reports of extreme disgruntlement among historians on open access, some of whom see open access as a threat to academic freedom, it was heartening to see others adopt a more constructive and indeed innovative approach.

The second was an excellent blogpost from Stuart Shieber, a key OA advocate at Harvard University, who laid out a clear case for the adoption of open access by scholarly societies, many of which are troubled by the perceived loss of subscription income from their journals. Shieber’s argument is clear and insightful and I can’t recommend it highly enough. It has already elicited a reaction from Kent Anderson at the Scholarly Kitchen that, to my mind, contains more mud than real criticism.

Onwards.

As part of the process the committee also issued a call for evidence from other interested parties. I have seen submissions from Mike Taylor, Ross Mounce and Heather Morrison and am sure there may be others. Before contributing I waited to hear what line of questioning the committee would take with Dame Janet; to avoid likely repetition, I tried to use my remarks to pick up on points that had emerged during that first session.

The complete submission prepared for the committee is laid out below (slightly reformatted to make it more blog-friendly).

SELECT COMMITTEE ON SCIENCE AND TECHNOLOGY 

Response to the 
Call for evidence on Open Access

18 January 2013

I.  Who am I and why am I responding to this call?

1. My name is Dr Stephen Curry and I am a Professor of Structural Biology at Imperial College London (though I am writing today in a personal capacity). I have been an active research scientist for around 25 years and published over 80 peer-reviewed articles.
2. To fulfil a sense of duty as a publicly-funded researcher, I also write regular science blogs at Occam’s Typewriter and for the Guardian Science blogs. Over the past twelve months have written extensively on the issue of open access (OA) from the perspective of a working academic.
3. Through this I have developed a useful dialogue with many interested parties, including academic colleagues, publishers and funders and a reasonably good understanding of many of the issues that are intertwined with open access. Last September, for example, I summarised the key outstanding questions for UK policy on open access in the light of the Finch Report and the new RCUK policy. I have also written about the severe problem of the over-reliance of journal impact factors as a measure of prestige, which is retarding the uptake of open access scholarly publishing. The committee has my gratitude and sympathy for taking up this important but complex topic.
4. I know that other academics have responded to this call so I don’t wish to repeat points that are likely already to have been made but I watched the committee take oral evidence from Dame Janet Finch on 15th January and was struck by some of the lines of questioning that were taken up. I wanted particularly to respond to these. My comments address the four issues to be considered by the committee’s inquiry but do not map neatly onto them because they are interlinked.

II.  Dame Janet’s introduction – the question of sustainability and support for OA through article processing charges (APCs)

1. In her opening remarks Dame Janet described one of the criteria for success in meeting the brief of the working group would be to come up with recommendations that would be financially sustainable for publishers and universities.
2. This statement does not quite make explicit — as Dame Janet did when speaking to Research Libraries UK in November 2012 — that part of the brief given to her working group by the Minister, David Willetts, was not to damage the publishing industry. The question is what is meant by ‘damage’ in this context and how far did consideration of this constraint affect the resulting recommendations (particularly in view of the fact that publishers were represented on the committee).
3. It is in most stakeholders’ interests to ensure that the processes of scholarly publication can continue without significant disruption during the transition from subscription-based access to open access. Publishers and universities therefore have some common interests, but it should not be forgotten that ‘financial sustainability’ will mean different things for the different stakeholders. While publishers quite understandably look to maximise profits, universities (and research funders) should be seeking to get the best value for money from the taxpayer-funded research budget. The large profit margins of some publishers (of the order of 40%) suggest that there is room to extract better value from the current spend on scientific publishing.
4. Research funders should seek to extract this value and savings on total publishing costs should be seen as an important driver for the move to OA. Although there are cost implications in the transition, in the long run, economic forecasts suggest that gold open access will be cheaper. I would argue that some loss of publishers’ profitability due to the emergence of a more active and transparent market with the rise of OA publishing is not a type of damage that should concern us. Rather it is something that should be allowed to occur as the consequence of technological and cultural changes.
5. Nevertheless, there are concerns that the emphasis of Finch and the RCUK policy on adopting a preference for gold OA will make the transition period unnecessarily expensive. The committee will no doubt have received a spectrum of views on that point — much debated among OA advocates in the last few months. It is problematic but one thing that should not be overlooked is that RCUK has made it clear that researchers can use green OA repositories to meet the terms of its new policy. These are free to authors but typically involve embargo periods of 6-12 months before the published research is freely accessible, and rely on the continuation of journal subscriptions.
6. My view is that the more costly route of preferring gold OA is a worthwhile investment because it offers spending power that can be used as leverage to allow publishers (and learned societies) to wean themselves off subscriptions. It also provides funds to encourage the development of new, more innovative OA journals that will create a more vibrant market, a move that is necessary to help drive down the costs of the article processing charges often required to publish via gold OA.
7. There is a risk that by providing funds for gold OA, the UK is locking in advantages for existing publishers and locking in current, relatively expensive APC pricing structures. However the rise in recent years of new OA publishers and journals, such as the Public Library of Science (PLOS), eLife and PeerJ, suggests to me that there could well be strong competition in the OA market. However, that strong competition will only happen if researchers are directly exposed to the cost implications of gold OA. This needs to be a feature of the implementation of the new RCUK policy at university level.
8. A related question that came up in the evidence session with Dame Janet. Lord Rees expressed concern about the administrative burden on universities of managing the funds allocated to them by RCUK to pay APCs. To an extent, this administrative burden already exists since several universities (including Imperial College) already have OA funds set up and have established simple procedures whereby staff can apply for monies to cover APCs. These will have to expand as the RCUK policy is rolled out but I don’t see the administrative problems as insurmountable. Indeed the expansion of these procedures is an opportunity to ensure that researchers are involved in spending decisions. Discussions and debates about the levels of APCs at the researcher level will help to create a transparent market that could exert downward pressure on prices. (One problem with the present situation is that researchers are generally ignorant of subscription costs, and this has in part contributed to them spiraling to unsustainable levels).

III.  Will a move to open access affect the prestige of UK research?

1. A more difficult question relates to the role of publishing in establishing and maintaining the prestige of researchers and UK institutions. This is an issue that was raised by more than one member of the Science and Technology committee on 15th January in questions about the impact of the UK policy on OA of our institutions’ international rankings. In particular there was concern that limits on university funds to cover APCs would prevent researchers publishing in ‘top journals’.
2. These concerns are overstated and misguided. They are overstated firstly since, at least at present, many ‘top journals’ (Nature, for example) enable authors to publish via green OA at no additional cost. Secondly, the entry of leaner, more innovative OA journals at the top end of the market (such as eLife) will reduce costs.
3. The concerns are misguided because the academic community has come to use journal impact factors as an easy proxy for the quality of a particular piece of research (or a particular researcher) when it is no such thing (4). Studies have repeatedly shown that the distribution of citations attracted by different papers in any given journal is extremely skewed. On average only about 15% of papers in the journal get large numbers of citations, while most are cited only rarely if at all. This pattern is maintained whatever the ranking of the journal. What this tells us, and it has been widely known for many years, is that impact factors are a poor indicator of the quality of an individual research paper.
4. Unfortunately, the widespread reliance on impact factors as measures of the quality of a paper, a researcher, or even a university has created an unhealthy situation that puts excessive power in the publishers of established titles. Until the research community can break its reliance on the impact factor proxy, and focus instead on evaluating and giving credit for the published work itself, the ‘top-ranked’ journals will be allowed to charge excessively high APCs.
5. This is a deep-seated cultural problem within the research community. It is being eroded but only slowly. However, the drive for open access will enable the adoption of article-level measures of quality because, by making the research literature more widely available, more people will be able to access and evaluate it.
6. Pre-publication peer-review should be maintained in an OA world in my view, as a check on the technical quality of the work that is being reported. But as the skewed citation distributions for the papers within any journal has shown repeatedly, pre-publication review by just two or three reviewers is unreliable as a determinant of the ultimate impact or significance of a given piece or work. For that evaluation to take place the research must be disseminated widely (and hopefully rapidly) and the response of the research community to it captured and reported.
7. It also needs to be borne in mind that the primary function of publishing is to share knowledge with the research community, businesses and the public; contributions to this function should be incorporated into measures of prestige rather than simply looking at the name of the journal.

IV.  How should the Government address the concerns raised by the scientific and publishing communities about the policy?

1. There is already some provision for this. The Finch group has been charged with conducting a review of the situation within the year (though I’m not sure a definitive timetable has been published), while RCUK has committed itself to reviewing its new open access policy within two years. It is to be hoped that these reviews will offer scope for input from the research community and other stakeholders.
2. However, there is one particular concern that I would like to raise. A key question for the UK (and one that I would like to see the committee put to Mr Willetts) is what impact its choice of a policy that relies heavily on gold OA will affect international cooperation on open access? This is a concern since most other research-active nations, including the US and the EU, appear to have a preference for green OA routes. It seems to me that this policy divergence risks greatly prolonging the process of transition to a system of scholarly publishing that is free from subscriptions and entirely supported by APCs. Mr Willetts has stated publicly that he would be discussing open access policy with ‘colleagues’ within and beyond the EU. It would be helpful to know what progress has been made in these discussions.

Through enforced inactivity the days have blurred. Still, I know the year’s end is upon us and sense that around the corner the unseen work of the year to come is taking shape. I don’t yet have the strength to face that low beast, but let me try to gather myself first by looking back on a year that is about to close. I may be drifting unsteadily through the holidays but perhaps I can find a hold on the tiller.

Let’s keep this short. Twenty-twelve was, more than anything else, the year of open access. I picked up on the issue back in January, incensed by Elsevier’s machinations over the Research Works Act, and haven’t let go since. It has been a long learning curve as, over the course of 32 blog posts, I covered debates, speeches, the Finch Report, the new RCUK policy and the ongoing ructions about how exactly it is going to be implemented. Along the way I wrote two of my most popular blogposts — they are still attracting traffic to this day — the first about an argument over PLOS ONE and the second, an impassioned plea for the end of impact factors.

These blogposts stimulated a valuable discussion in the comment threads, from which I learned a great deal, but they also triggered invitations to contribute articles, to be interviewed on radio and TV and to speak at meetings and debates. Without ever intending to I have become a sort of mouthpiece. That’s not to say that my views are mature and ripe for dissemination to the wider world; they are very much still in gestation. I frankly don’t know how the UK’s declaration for gold OA (while still allowing green) will play out. Resistance to Finch has, if anything, grown since the release of the working group’s report in the summer and the RCUK’s announcements of its mode of implementation; the latest salvoes have come from the Humanities and Social Sciences, history journals in particular having set themselves against the winds of change.

The open access issue may seem vexed in many quarters but I remain optimistic. I never expected the Finch working group — and nor I suspect did they — to be able to produce proposals that would satisfy all parties. They have certainly shaken things up a bit and nobody knows how they will settle out. Only one thing is clear: there is more work to do and there are more arguments to be had in 2013.

Open access wasn’t the only science policy issue on my mind in 2012. The ongoing travails of the declining UK science budget were never far away, nor was the persistent question of what sort of science merits public funding (answer: all sorts). I was delighted in November to be elected onto the Board of Directors at the Campaign for Science and Engineering and look forward to continuing my education at the interface between science and politics (a frictive junction that has produced considerable heat and just a little light of late).

I was more delighted still to witness back in May the introduction to parliament of a bill to amend the law on defamation in England and Wales. This was the culmination of the libel reform campaign, which had kicked off in 2010 as Simon Singh was fighting off the erroneous claims of the scientifically illiterate British Chiropractic Association. The campaign has wide, indeed cross-party, support but the reform is not yet a done deal. The bill is presently at the committee stage in the House of Lords but still needs work to bolster the public interest defence. The government has recently offered some concessions on this but the campaign would like to see more wrung out before finally, hopefully, the new law is enacted. Watch this space.

There is life beyond campaigning — thank goodness. In August Occam’s Typewriter set up an Occam’s Corner shop on the Guardian Science Blogs. Though terrifying at first — and though I’ve yet to find the right voice for this new venue and audience — it’s remarkable how quickly you can get used to things. I am enjoying the new surroundings and have started to have a little fun, at least with Simon Jenkins and DNA.

I had even more fun at Matilda — I’m not big on musicals but it was sublime — and at the Edinburgh Festival where my personal highlights (apart, of course, from my daughter) were Pappy’s Last Show Ever and Tony Law’s Maximum Nonsense (follow the links and you can catch them too). On the scientific front, Jim Al-Khalili’s Order and Disorder was the most wonderful thing on television, while the transit of Venus gave me a moment of purest astronomical joy.

And that is quite enough words for now; I think I can see the outline of the shore.

And I like carbon. In fact, I’ve decided that it’s my favourite element. I’ll tell you why in the short video below, which is part of the Royal Institution’s 2012 Advent Calendar.

Please take a little time to explore the rest of the calendar. Each day a different person – usually a scientist or broadcaster (or sometimes both) — picks out their favourite element, and it is making for a lovely and informative series if short films.

So far my favourite is Helen Czerski’s introduction to calcium on Dec 5th (which has echoes of an earlier talk given by Thomas Henry Huxley, something of a hero of mine and a regular performer in his day at the Royal Institution).

But I was also touched by Martyn Poliakoff”s revelation of the origin of his affinity for Sodium the following day.

Mark and I had already met, at an open access debate at Imperial College back in September but this was the first time I had encountered Dame Janet. Having spent time reading her report back in the summer, I was pleased to discover that she had in turn read some of my output on the topic of OA.

The conference was a good opportunity to talk to both Mark and Dame Janet and to get a better insight into the thinking behind the Finch report and the new RCUK policy. Some of the more colourful remarks made are off the record, I’m afraid, but there is still plenty of information to be gleaned from the presentations made, which were recorded and have been uploaded to YouTube (thanks to the good offices of RLUK’s Melanie Cheung).

Dame Janet spoke first (video)— and was very open about the remit and process of her working group. I was intrigued to hear her confess that she hadn’t known much about open access before being asked to take charge of the committee (though she was briefed in detail by Phil Sykes and Paul Reynolds, chief librarians at Liverpool and Keele Universities respectively). I guess the government was looking for an academic without a preconceived agenda to lead development of new policy recommendations.

Dame Janet being open about open access

I recommend that people listen to the talk — or at least the opening remarks — to get a feel for Dame Janet’s sense of her committee’s mission and the response to it. It adds the human (and sometimes humorous) dimension to a debate that has at times been fractious. I wish she would speak more often about the thinking behind the report since it would open the conversation on open access to a broader audience. Her talk was revealing in ways that the report is not: in particular Dame Janet was up front about the responsibility imposed by the working group’s remit not to damage the publishing industry. This is something that many had detected in reading the report, but I had not previously heard it stated so boldly.

Of course such a provision makes some sense for a government-led initiative, keen to protect profitable businesses in the UK, but of course there are tensions between that and the need to secure good value for money on public spending and to see through the project of making open access work on a global scale. One has to wonder whether the remit not to do damage led the group to underestimate the impact of technology-driven changes that are by their own momentum making paper-based publishing obsolescent and inducing a fundamental re-think of the nature and process of scholarly publishing.

No-one seems quite sure, including Dame Janet, who has been charged with reviewing the situation in about a year’s time. For this reason continued constructive engagement in open access issues by all stakeholders will be important.

Next up was Mark Thorley (video) who laid out the thinking behind the interpretation of the Finch report that is enshrined in the RCUK’s new policy on open access (already covered in his blogposts). Mark was able also to spell out some of the details on the financial provision — announced last week — and how it will be allocated to universities. The approach is pragmatic, reflecting an expectation that there will be significant but not immediate growth in uptake of gold OA routes once the policy goes live in April next year. The plan is to ramp up funding steadily to cover predicted article processing charges (APCs) of 45% of all RCUK output in 2013, rising steadily to cover 75% by 2017. The expectation appears to be that by then the remaining 25% will be made OA by the green repository routes.

The RCUK will operate a ‘light touch’ regime of monitoring how the money is spent, again reflecting a degree of pragmatism (rooted sensibly in the remaining uncertainty about how things are going to pan out). However, there are some important points to take on board. Though the details of mechanisms are yet to be finalised (RCUK held a meeting with universities to discuss this on 13th Nov), Mark made a clear acknowledgement that monitoring of compliance with OA policies had been weak in the past and that this would be much stronger from next year, an assertion that applies irrespective of whether RCUK funded research is published by gold or green OA routes. Although the preference for gold OA remains, he again made it clear that researchers are free to choose green OA routes for their work if this is what they (and their institutions) would prefer to do.

A further point of interest is that RCUK will gather data on what different institutions are paying for their APCs and, crucially, will publish this information so it can be used in negotiations with publishers to reduce subscription charges. This should help to ease the cost of the transition by focusing attention on the need to couple increased use of APCs in the UK with lower institutional subsciptions.

Finally it was my turn to speak, to give a researcher’s perspective. I won’t go into details since my arguments have already been developed at extravagant length on this blog. For newcomers, it will probably be quicker to sit through the whole 30 minutes of my talk. Apologies. Alternatively, skim through the slides.

It’s good to talk. I was glad to have the opportunity to address research librarians. They are an important group of stakeholders in the messy business of open access who are the natural and expert allies of academics. In many institutions around the country they have already begun to think about how to implement the new policy. One thing that I did try to impress on those present was that, whatever procedures are to be put in place, we must ensure that researchers are exposed to the actual costs of publishing so that they can make informed decisions about the best route to open access for them. Only with visible costs will we develop a functioning market in scholarly publishing that pays sufficient attention to the proper balance between price and quality.

But the duty of implementation falls not only to librarians. Researchers are just as responsible for making OA work but, unfortunately, many have yet to pick up this issue with serious intent. They are too busy and too inured to traditional modes of operation (and impact factors). But we need to talk to more of them so that this issue can break out of library committees and out of the blogosphere. Ideas for how to do that are most welcome.

The start of open access week

From 22-28 October 2012 the world celebrated open access week and along with many others I played a part in getting the message out, using a mix of traditional and new-fangled ways.

My week kicked off with a Monday-morning blogpost at Occam’s Corner about a nicely timed paper from Mikael Laakso and Bo-Christer Björk showing that gold open access publications now account for 17% of the approximately 1.7 million research articles published in 2011. This figure, which is based on thorough and clearly-explained sampling methods, is significantly higher than previous estimates and suggests that progress in open access has reacher a higher water-mark than anyone realised.

That afternoon I participated in the Opening Research and Data meeting at Birkbeck (jointly organised by LSHTM, Birkbeck, LSE, SOAS and City University) to talk about the shape of the open access landscape following the Finch Report and the announcement of the new RCUK OA policy. There’s a video of the entire proceedings and, for the time poor, a nice summary at the City Open Access blog; (my slides are also available on Slideshare). I was particularly interested — and pained — to hear of the experiences of Antonio Gasparrini, an early career researcher who talked about the travails of trying to balance his budget with the systemic demands that he comply with OA and publish in journals that would enhance his career prospects, by which of course he meant journals with higher impact factors. I would much rather he took the advice of another speaker, Melissa Terras, who showed how a modicum of social-media driven self-promotion can get your work the attention it deserves.

Last to speak was Ben Ryan of the RCUK who sought to further clarify their new OA policy, particularly on the point that it is up to authors and institutions to choose whether to publish via green or gold OA routes (something I covered recently). In the questions that followed Ryan was also at pains to emphasise that the research councils will be putting in place measures to ensure that their funded researchers comply with the new policy, something they have done only half-heartedly, if at all, in the past. These compliance rules will apply whatever the colour of the OA route selected. It is to be hoped that they will be forceful since, as Stevan Harnad and colleagues showed this past week, the strength of green OA mandates is strongly correlated with rates of deposition.

However, Ryan was not in a position to spell out the full details of the new compliance procedures, or to give a clue as to how much RCUK will be allocating to fund its new policy which, as Ryan repeated, allows green while retaining a strong preference for gold OA. Those details should be forthcoming ‘this Autumn’.

More on the RCUK OA policy

The following Wednesday Mark Thorley sought to add further justification of the organisation’s preference for gold on the RCUK blog. It is refreshing that RCUk is engaging in this open mode of communication but the case presented was not wholly convincing. The case rests on the principle that “ideas and knowledge derived from publicly-funded research must be made available and accessible for public use, interrogation and scrutiny, as widely, rapidly and effectively as practicable”. From this principle flows the idea that payment of an APC to publishers ensure that the final peer-reviewed and formatted version of a research paper is made freely available from the date of publication and under a CC-BY licence that allows anyone or any organisation to re-use the content (so long as it is attributed), even for commercial products.

However RCUK seems to lack some conviction in either the worth of the CC-BY licence or in its right to enforce such a licence on the content of research published through the green OA route, since the policy allows RCUK-funded researchers to choose green or gold OA routes. For those who choose green OA, the RCUK feels it can only demand a CC-BY-NC licence. This still permits access to all and re-use by non-profit organisations, albeit after a maximum six-month embargo. For many users of the research literature, such a condition may well be practicable but I wonder how much confusion might be sown by justifying the declared preference for gold on the bases of a solemn principle.

Another oddity of the policy clarification is RCUK’s notion that potential readers might not be able to tell if the author’s post-review version of a paper in a green OA repository has the same content as that formatted for the journal where it is published. This is cited as another motivation for gold OA but to my mind under-estimates both the intelligence of readers and the ease with which authors can simply add a statement to the deposited version that makes its identity with the journal version clear. It also seems to forget that most publishers require authors to clearly link the deposited post-print to their journal version.

As is clear from the comment thread underneath Mark’s blogpost, I am not the only one with ongoing questions about RCUK’s direction of travel. Given the residual dissatisfaction, I don’t think this conversation is quite finished, but it is good at least that the channel of communication remains open.

Is the UK really leading on OA?

The rumbling confusion engendered by the unfolding of the RCUK policy stood in contrast to the announcement of Ireland’s new open access policy which is very green in hue, with gold being considered unnecessary but permitted should researchers or their funding agencies wish to pay for that option. The Irish policy declares itself to be based on ‘best practice’ and is compliant with policy statement from the EU, the OECD and the revised recommendations of the Budapest Open Access Initiative, which were announced in September this year. Ireland’s alignment with the positions of these international organisations raises a tricky question about the UK’s claim to be in a world-leading position on OA policy, a claim re-iterated by the RCUK’s Ben Ryan at the Opening Research and Data meeting.

David Willetts, the minister for universities and science, who deserves credit for driving the UK policy on open access, said back in May that he would be having discussions with international partners. He rightly recognises that fully operational open access requires global cooperation. But we have yet to hear any reports of the progress of those discussions and given the various developments that have occurred in the last several months and the growing sense that the UK may be on a different track from most other nations, it would be good to have an update. Unfortunately, unlike Mark Thorley at RCUK, Mr Willetts is not a blogger.

The end of Open Access Week

On Thursday of Open Access Week, I tried the mod-con that is known as a Goole hangout and chatted online about OA in a PLOS-hosted event organised by Cameron Neylon. From my London office I discussed the most important events of the year gone by with Cameron in Cambridge, Mike Carroll in Washington, Heather Piwowar in Vancouver and Marina Kusko in San Fransciso. Using a technology unthinkable 10 years ago, we discussed changes to the publishing landscape that themselves used to seem unthinkable. Such is the power of the web.

You can see the whole conversation below, though be warned: it’s nearly 50 minutes long. (Such is the power of the web. )

The next day I participated in an even more modern form of web-based conversation — a live twitter chat on open access organised by BioMed Central to address the questions:

Q1 – What are the main factors that have led to the steady growth of OA publishing?
Q2 – How do you think this trend will develop over the next decade, and explain why?
Q3 – What challenges does the growth of OA publishing face in ensuring that it reaches its full potential?
Q4 – Where will the funding for OA publishing come from?
Q5 – Do subscription journals offer benefits that OA journals do not?

It was fairly fast-paced but, as you might expect for something twitter-based, restricted to fairly pithy remarks. You can track the edited highlights in the Storify version (worth following all the way to the end, if only to read OpenAccessHulk’s closing remarks).

Those questions reflect both the continuing growth of OA and the continuing uncertainty attached to it. Nobody can quite figure out what the future will look like. Of course, Open Access Week was a good opportunity to think about the issues some more but even though the week is over, the debate is far from done and there have been some interesting contributions and observations since the week came to an end.

Publishers: more bad and some good behaviour

Last week the Times Higher Education magazine ran a commentary by Simon Lilley of Leicester University highlighting not only the comfortable operating profit margins (19-41%) of several major publishers (whose gross profits on journals are reckoned to be even higher) but also the fact that Informa PLC, owner of Taylor & Francis and Routledge imprints, re-organised itself in 2009 to become a Jersey company that is domiciled in Switzerland, apparently to save itself £12.3 million in tax. I guess that makes economic sense but such actions will earn these publishers no credit with the academic community in the UK. Elsevier is still struggling to restore its reputation following the debacle of the US Research Works Act.

The actions of Informa only serve to remind us of the hard-nosed attitude that some companies take to the business of publishing, however much they like to portray themselves as partners in research. They are out of kilter with the public’s sense of justice, already riled by news of the tax avoiding strategies of the likes of Starbucks, Apple and Amazon. In this case the sense of unfairness is exacerbated by the fact that Informa’s tax avoidance deprives public coffers of funds that help to pay a large fraction of the UK share of the publisher’s profits!

Not all are quite so hard-line. Deborah Kahn of BioMed Central articulated a more reasonable view of the value of the service that publishers can bring to researchers. This is clearly a move in the right direction, though all publishers still need to be mindful of charges and profits. Some in the academic community are not prepared to wait for all publishing companies to change their spots and want to take matters into their own hands. Deborah Shorley, Director of Library Services at Imperial College, argued a passionate and personal case for universities to reclaim control of the publication of the work that their researchers do, hoping that they might at the same time re-discover their original purpose. Physicist Peter Coles is doing just that and plans to launch a new open access astrophysics journal, as yet unnamed, in January 2013.

Change is coming, and at all levels. Nature Publishing Group announced during Open Access Week that it is embracing article level metrics for its Nature-branded journals. The metrics will include citation data and page views, as well as mentions of the article on social media sites. This move isn’t directly relevant to open access but it may help in swinging the tide against the mis-use of journal impact factors, which continues to act as an impediment to publication in open access titles.

And finally — the view from Africa

And finally, lest we get carried away that all the activity in Open Access Week was focused on the UK, the EU and the US and the debate about gold vs green, let me point you to this informative and poignant blogpost from Eve Gray in South Africa, who highlights out the particular challenges faced by African researchers. Controversially perhaps, she argues that green OA, favoured by many campaigners in the developed world, may not be the best solution for Africa. It is an excellent, heartfelt piece that brings an important perspective to the debate (my emphasis in bold):

…at heart [the debate] takes us back to the question of whether we are seeking access to or participation in the production of global literature. Which policy path would most effectively give voice to research from Africa, largely silenced in the current system? Access to world literature is also important, but is inadequate on its own, risking perpetuating a neo-colonial dispensation that casts the dominant North as the producer and the developing world as the consumer of knowledge.

I have come to think that the green/gold debate is in fact a distraction from dealing with more insidious issues in our research publishing systems. These include the dominance of journals at the expense of other forms of publication; the almost universal adoption of the ISI and its Impact Factor as the basis for recognition and reward; and, most insidious of all, the marginalization of great swathes of global research through the implementation of this commercialized ranking system.

I urge you to give Eve’s post the time and attention it deserves. It is a powerful reminder that open access is a global necessity and that the issue has even more dimensions than most of us realise.

Presented by Prof. Jim Al Khalili, the first episode of Order and Disorder is devoted to the slippery concept of Energy. Tracking through history Al Khalili tells the tale of how the emergence of the all-conquering steam engine focused the minds of scientists on the question of how heat was being converted to do useful work and collided inevitably with the even more abstruse notion of entropy.

As I watched, my excitement and admiration grew because the programme steadfastly refused to shy away from the difficulty of the topic. It held its nerve to explore the discovery that entropy emerges naturally from the fact that the universe is made of atoms. I have never seen the subject unfurled so adroitly before.

Better yet, this is a beautiful film. Too often science on TV is ill-served by the visual nature of the medium. The subject becomes subservient to the images used, too many of them being a wrong and therefore distracting choice, or worse — clichés. Here instead there was an artful unity of the visuals and the science. The film includes a visit to the Crossness steam pumping station in south east London, where the camera pans lovingly over the decorative detail that the Victorians lavished on their cathedral of power. There is very good use of computer graphics to illustrate the dispersion of heat through atomic motion and a sequence of great fun and originality in which Al Khalili sketches out an equation on entropy using a hairdryer. But my favourite shot is of condensation dribbling blackened tracks from a statement of Boltzmann’s entropy equation, written there moments before in marker pen.

Regular readers of this blog will know I am a fan of thermodynamics (and atoms). Like CP Snow, I wish that more people might share this enthusiasm and, thanks to Order and Disorder, they can.

I hope the BBC might leave it on iPlayer permanently.

*I originally thought the show was only available until Tues 23rd but mis-read the information on iPlayer.

Update 29-10-12: The second episode, which discusses information — the flipside of entropy — is just as good and can be seen on iPlayer until Sat 3rd Nov.

You can listen to the entire discussion or, if pushed for time, follow the tweets in Jon Tennant’s Storify version, read Paul Jump’s report in the Times Higher Education or have a look at Ian Mulvaney’s thoughtful summary.

Preaching to the choir? (Photo by Anna Zecharia)

There is no need for me to rehash what was a very useful discussion but I did want to pull out the points that have struck me most on the night.

I was a little surprised that Thorley started out rather defensively but perhaps that’s because he had already spent the day defending the RCUK policy at another meeting and was feeling a little bruised. If that is the case then I take my hat off to him for having the stamina to stay the course for the evening at Imperial.

For me the most important piece of information that came out of the discussion was the clarification of the RCUK policy. There had been confusion over the options available to authors funded by the Research Councils when trying to publish in journals that offered gold or green OA options, particularly if the gold OA route required payment of an article processing charge (APC).

The guidelines published back in July (PDF) were interpreted by many (myself included) to mean that if a journal offered gold or green OA options, the author would be obliged to opt for the gold route.

However, Thorley made it clear that interpretation is incorrect. He repeated the clarification on the RCUK blog the following day so, to make sure I get it right, I will quote from that (with my emphasis in bold):

“If the journal they want to publish in only offers policy compliance through a Gold route, they must use that journal’s Gold option. If the journal only offers compliance through the Green route, the author must ensure that a copy of the post-print is deposited in an appropriate repository – for example, UKPMC for papers arising from MRC funded research. If the journal offers both a Gold and a Green route to compliance (and some journals already do this), it is up to the author and their institution to decide on the most appropriate route to use. And, if a journal offers neither a Green nor a Gold compliant route, it is not eligible to take RCUK funded work, and the author must use a different, compliant, journal.”

Thorley assured us that the clarification would also appear in the more detailed guidelines to be published by the Research Councils, which should appear within a few weeks.

The key point here is that, although RCUK has a clear preference for gold OA — motivated largely by their desire to ensure that a CC-BY licence can be attached to papers to ensure free re-use and text mining — green OA routes are clearly available to authors.

The emphasis on gold OA has been criticised in several quarters (see this earlier post for a summary) but Thorley was emphatic that no modifications would be made to the policy in the short term. He did promise that there would be a review within a year or two, preferring to see the policy statement as the start of a new approach to open access in the UK, rather than an endpoint.

That line won’t satisfy everyone but unanimity of strategy was never likely to be achieved since the OA movement is a broad church, subject to some of the same tensions found within real churches.

On the night the discussion moved on to more technical questions about how the policy would work in practice. Research institutions in receipt of RCUK funding have started to grapple with those questions, especially those in receipt of the £10m sweetener handed out in September by science and universities minister David Willetts.

I very much hope that, whatever mechanisms institutions adopt, they will be visible to authors, since their exposure to price constraints will be key to driving down the cost of APCs. It would be a mistake if universities devised systems (like the current management of journal subscriptions) that are largely hidden from their academic staff. Thorley reported that RCUK will be monitoring how OA funds are spent, which is good: this should also help put downward pressure on costs.

One of the fears associated with the  RCUK preoccupation with gold OA is that it may lock in a pricing level that preserves income streams to publishers. We need to make sure that authors’ and institutions (and the government’s?) desire for value for money starts to exert pressure in the publishing marketplace.

And so we head somewhat uncertainly into the future. A particular concern is that no-one has any idea how long the transition period will take. If it becomes protracted, the excess costs borne by the science budget run the risk of undermining wider support for open access publishing. Within the blogosphere the move to open access might seem inevitable but it would be a mistake for proponents to assume that the wider research community shares all of their assumptions. I remain concerned that conversations about the value of open access to research and society at large are still not happening frequently enough among the key stakeholder: researchers.