One of these days — I promise — I will get back to writing about science. But a conjunction of tweets today brought to me three articles on open access that were interesting in different ways but curiously all seemed to point in a similar direction.
First, a post on the Scholarly Kitchen blog (h/t Alicia Wise) from Rick Anderson, who is the Associate Dean for Scholarly Resources & Collection at the University of Utah. His post was a critique of the ‘Big Deals’, bundled or grouped journal subscriptions that large publishers sell to university libraries. They may have low ‘per article’ costs but inevitably mean that libraries have to subscribe to journal titles they don’t want. Big deals consume very large fractions of library budgets and restrict flexibility for changing subscriptions.
But the more interesting point was that the ‘big deal’ was a larger version of a long-standing practice of ‘Medium Deals’ by which he meant ‘journals’. To get access to an article of interest you have to subscribe to the whole journal. Anderson argues that this made sense when the literature had to be printed to be distributed; but this is no longer the case now that distribution is online. He doesn’t quite come out and say it, but are we on a trajectory that leads to the abandonment of the journal form?
There seems no immediate danger; even relatively new and online-only journals, such as PLoS ONE, have a journal identity. But what does that mean in the absence of a physical object that looks like a printed journal? I suspect the dissociation of the concept from the thing itself may weaken people’s habituation to the form.
There is some evidence for this from the second interesting article I came across today, a recent preprint in the ArXiv (h/t @figshare) that claims to detect a “weakening relationship” between journal Impact Factors and individual papers’ citations. To quote the abstract:
Historically, papers have been physically bound to the journal in which they were published but in the electronic age papers are available individually, no longer tied to their respective journals. Hence, papers now can be read and cited based on their own merits, independently of the journal’s physical availability, reputation, or Impact Factor. We compare the strength of the relationship between journals’ Impact Factors and the actual citations received by their respective papers from 1902 to 2009. Throughout most of the 20th century, papers’ citation rates were increasingly linked to their respective journals’ Impact Factors. However, since 1990, the advent of the digital age, the strength of the relation between Impact Factors and paper citations has been decreasing. This decrease began sooner in physics, a field that was quicker to make the transition into the electronic domain. Furthermore, since 1990, the proportion of highly cited papers coming from highly cited journals has been decreasing, and accordingly, the proportion of highly cited papers not coming from highly cited journals has also been increasing. Should this pattern continue, it might bring an end to the use of the Impact Factor as a way to evaluate the quality of journals, papers and researchers.
This doesn’t amount to a prediction of the end of the journal, (though there is some prospect of reducing the disturbing hold of the Impact Factor over academic careers). But it does suggest that it is the paper and not the journal that may be the focus of interest in future.
Traditions, like familiar coastlines, are slowly being eroded. The standard journal faces new competition from the disruptive impact of the web. And it is not just the advocates of open access who are saying that. The third piece flagged up to me on Twitter today (h/t @GaviaLib) is a sober analysis by Jared Woodard, a financial analyst, of how the rise of viable open access options is presenting real challenges to Elsevier.
His analysis is worth reading in full but to pick out just a few quotations:
Elsevier’s upside potential looks capped, since there is no room for top-line expansion given tight university library budgets, and no room for cost-cutting in an industry where the labor of academic researchers, editors, and peer reviewers is provided, literally, for free.
Social media and improving technology have allowed opposition to rent-seeking by publishers to become more coordinated and widespread. The public relations disaster and boycott triggered by the Research Works Act (RWA) provides an excellent example.
The best argument that the company has put forward to defend their business model is that, in exchange for all of the free labor that makes the business so profitable, Elsevier provides academics with the “high-impact” journal brands that committees want to see on the CVs of tenure applicants. Until recently, the only response from the open access movement was to point out that rentier capitalism is unjust. Today, the best response is that high-impact open access journals exist as a viable alternative to the old model.
These are uncertain times for publishers — and academics. Change is coming, and perhaps more than any of us is bargaining for.
And to help push things along, I have signed the White House petition on open access. Come on in; the water’s intriguing.
Interesting stuff, Stephen.
I´m definitely in favour of a shift to OA, but journals are so much more than just an IF.
Subject specific journals play a major role in the filtering process, which makes my life so much easier in terms of knowing where to go to find relevant, reliable research. E-mail TOCs are a really useful tool in this sense.
I also submit to specific journals because I know the Editors will be able to find good reviewers and be knowledgable enough in my subject area to adjudicate any disagreement between author/reviewers. Some journals will prefer different aspects of work as well, e.g., favouring theoretical insight or empirical advances. Scientific/Learned Societies may have other goals in mind for their journals than just climbing the IF ladder.
More general journals have also played an important role in exposing readers to ideas from different fields, although this important byproduct might be disappearing as we change our reading habits.
IF has become a pretty easy target to attack. It´s losing (an undeserved?) credibility all the time. But that is not the only thing that journals represent. It´s important not to lose sight of this, and while I´m happy for the reliance on IF and its associated elitism to break down, I think there are so many other useful features of journals that I´m not quite ready to give up on them as a useful concept yet.
If you’re right that the sustained assault on Impact Factor’s wholly undeserved credibility really is having the effect of reducing or maybe eliminating its distorting effect on science, then — well, hurrah! That alone would be enough to undo maybe half of the damage that the journal system does.
Hmmm, having had a quick look through the arXive manuscript you linked to, I´d be very wary of accepting their interpretations at face value.
They may ultimately be correct, but the Methods they use in the analysis of how the relationship between IFs and citation rates changes with time are not clearly outlined, and they certainly don´t appear to use current best practice in time-series analysis. For example, why do they break up the series in the specific way they do in Fig 1, generating 3 different models across the whole time series (where only the final part of the series is shown to have a negative slope)? Is there any support that this approach is statistically more valid than just using a single model (or 5 different models) to describe the whole time-series? I may be wrong, but it looks suspiciously like the specific time segments they chose were “fitted by eye”, which is neither reliable nor reproducible.
It seems their conclusions are very heavily influenced by this approach, and it´s at all not transparent from the Methods presented why they did it this way.
All this highlights another useful feature of peer-reviewed journals (which the arXiv is not). Whether it´s poor methodological practice, or just poor presentation, the interpretations of this MS are not supported by the data yet. (And apologies for briefly turning the discussion here into a reviewer´s report 😉
Mike, I’d definitely like to see maybe a Bayesian analysis where the break points are treated as parameters. The other thing they didn’t discuss is why is there an enormous jump in the correlation at the beginning of the 90s? That seems to me important to discusss to give credibility to the data and interpreation.
That said, the graph which spoke most to me was the one tracing the percentage of top 5% of articles (by citation) in top 5% of journals (by IF or proxy IF) which, although I accept “by eye” has a very strong inflexion point near the beginning of the 90s.
Sod staight lines and breakpoints, fit a spline to it. That jump in the 90s looks strange – I wonder if it’s an artifact of something odd in the 80s.
Whilst we’re being reviewers, on of the frustrating things about this paper is that they don’t tell us a lot about the data, so it’s difficult to get a feeling for what’s happening. For example, we know that the number of journals has increased, and my impression is that the number of citations per paper have increased. There’s also variation between subject areas (compare Figs 1-3), which might be informative. The bottom line is that the authors get a result and then fail to explore the data to see how far it goes in explaining the result.
If Fig. 4 is to be believed, somewhere between 20% and 40% were not published in either the top 10% or the other 90% of journals (by IF). I think this does undermine the paper somewhat.
Some important points on this thread. I just sent the following email to Lozano, Larivière and Gingras, the authors. Hope they respond.
Many thanks for posting your important manuscript “The weakening relationship between the Impact Factor and papers’ citations in the digital age” on arXiv:
You’re probably aware that it’s had quite a bit of attention on Twitter and blogs, and rightly so. But I wanted to draw your attention to this comment thread:
which I think raises some important points. It would be good if you could address them in that thread, or (better still) strengthen your manuscript by taking those comments into account as an informal peer-review.
Cheers Mike – I hope they respond.
I agree this was a rather speculative post Mike — more a sniffing of the air than anything else. But the Scholarly Kitchen and ArXiv posts were interesting new smells and I wanted to see what others made of them.
I take your point that some of the data in the arXiv paper looks a little suspect; I couldn’t figure out either how they justified the choice of fits to the data. There did seem to be consistent trends but you are right to emphasise this manuscript has yet to be peer reviewed (apart from by your good self).
Just to be clear, I don’t have a problem with the data (other than the accessibility issue Cameron points out above*). It’s the methods of analysis, interpretation and presentation that I’m more concerned about. Bayesian or Frequentist approaches coupled with Information Criteria comparison, are available (and standard in many fields) that would easily improve on what seems to be presented in that arXiv paper.
I’ve got a bit of a bee in by bonnet about journalistic reporting of arXiv papers** at the moment. I’m glad that e.g., the BBC are basing their science reporting on specific papers, not just on press releases, but the risks of publicising a science story based on non-peer reviewed manuscripts is too great in my opinion. It’s irresponsible journalism, especially with the Wakefield trauma’s effects still being felt (and that was peer-reviewed!).
But a discussion on the more general issue of journal presentation is certainly warranted. Can we improve the quality of our science by changing the way it’s presented? This goes way beyond the basic OA debate into exactly the sorts of questions raised here.
* This is also a fascinating point in terms of publication ethics and terms. If the authors aren’t allowed to give their data to others to re-analyse (never mind whether they want to share it or not, and many don’t), it may breach the terms in some journals’ publication policy.
** arXiv is a great resource and service. It just has to be used appropriately.
I agree that the BBC etc. should be careful when reporting on papers that have yet to be peer-reviewed. I’ve noted deviation from best practice before.
I was relieved on re-reading my post to see that I had been careful enough to refer to the ArXiv paper as a ‘preprint’ and to mention the result was claimed rather than demonstrated. Nevertheless, probably a good idea to explicitly state in such cases that the paper has not been reviewed.
For anyone unconvinced by pre-publication peer-review as the stamp of quality that marks a paper as worthy of reporting in the mainstream media, surely the Wakefield autism debacle will be interpreted as a reason not to put our trust in peer-review? In that case, the only peer-review that actually mattered is what happened after it was published (and reported).
Mike T: given the substantial extra story behind this particular case (allegations of fraud, serious financial conflicts of interest and substantiated accusations of unethical investigator behaviour towards minors, leading to Wakefield being struck-off by the GMC), this is not necessarily a good example of a failure of peer-review. I realise I brought that example up, but I don’t want it to imply that I mean that pre-publication peer-review is totally flawed. Sorry if I gave that impression.
The reviewers of Wakefield’s Lancet paper could not reasonably be expected to know these seriously problematic factors when evaluating the manuscript – an otherwise basic case-report study. Subsequently, his co-authors have asked to be removed from the author list and for the paper to be retracted when they found out the scale of the problems.
Further, the MMR media scare is more attributable to comments made by Wakefield in the press-conference that coincided with publication of his paper, rather than anything specifically contained in the paper itself.
Pre-publication peer-review is frustrating as hell sometimes, but works well most of the time. It can probably be improved upon, but there is not yet consensus on exactly how to do that. That doesn’t mean we should abandon it as a useful filtering mechanism at the moment though.
I didn’t mean to accuse the reviewers of the Wakefield paper of falling down on the job; almost the opposite, in fact — my point was (meant to me) that even when peer-review is done well, it can’t stop these things from happening.
Is pre-publication peer-review worth it? I’m genuinely on the fence. Unquestionably it gives huge benefits; equally unquestionable, it imposes huge costs. I’m really not sure which side of the fence I come down on. I guess I most like the PLoS ONE model, where reviews are entirely about asking “is this good science?”
I agree entirely!
Incidentally, I would say this is the sort of ‘standard’ that’s generally employed by philosophers (and perhaps other humanists) in peer reviewing a manuscript. If we had to agree on the “rightness” of a conclusion, I’m not sure anything would ever pass peer review. Instead, we ask, regardless of whether we agree with the conclusion, if the manuscript is well-argued, cites relevant literature (which is also controversial, in some sense), and so on.
A major difference between peer review in philosophy and science that would remain is that post-publication peer review would have more chance of settling the matter in science. We philosophers don’t make progress in the same way.
You make an interesting point, Stephen: “you are right to emphasise this manuscript has yet to be peer reviewed (apart from by your good self).”
If the journals disappear and are replaced by the article as the main focus, we’re going to have to explore alternatives to journal-run peer review. I tend to agree that the sort of comments Mike makes do constitute peer review. But we need some way to organize such reviews.
This is the most intriguing option I’ve found: http://www.peerevaluation.org/profile/profileID:qoqLmq3+rd8=. If you click on one of the papers, you’ll see that you’re given the option of reviewing it. You can make the review public, and you can include your credentials. Anyway, lots of nice features here. For an entertaining video introduction, go here: http://www.youtube.com/watch?feature=player_embedded&v=8omwsw8ibzw.
I suspect there will still be a gravitational pull of subject areas — which helps in the organisation of peer-review (as Mike pointed out in his first comment). My own work has spanned several fields so I’m not as mindful of targeting journals because I know the editors — most often I don’t.
The rise of PLoS ONE maybe shows the middle way — a mega-journal that has sub-sections by dint of the particular specialities of it’s editors.
I think this is one of the few criticisms I have about PLoS One: I find it very difficult to keep up with the massive amount they publish, across the fields that are interesting/relevant for me.
This at least partly reflects my own inefficiencies in filtering their content, but if they could offer clear subject sections in a TOC a la PNAS, it would certainly help me.
There’s definitely a market out there for people or services that act as editors to provide tailored sets of articles from PLoS — or, better, from all open-access publishers. I could happily see a weekly cross-journal ToC of papers relevant to my interests.
That could be done either by automatic filtering based on keywords that I specify (hopefully with some intelligence regarding synonyms, near-synonyms, etc.) or by human editors within particular subject areas.
The great thing of course is that because PLoS and BMC are truly open access, anyone can do this — whoever builds the best service wins, and potentially gets rich. (Could be a problem including CC-BY-NC journals, though: when the money starts flowing in, such journals might cause problems by claiming that the ToCs are a derivative work in contravention of the NC clause.)
You should know by now what the more vocal people in the OA debate make of the smells from the Scholarly Kitchen.
You wouldn’t be suggesting that the posts that go up on Scholarly Kitchen have more holes in them than a particularly ripe Swiss cheese, would you Bob?
Not at all. Merely that that’s how they are perceived in some quarters.
I find SK to be a mixed bag… thought Rick Anderson’s post was of the more thoughtful variety.
It is a mixed bag so far as the actual articles go: some astonishingly head-in-the-sand, some very percerptive. It’s in the comments that things always go wrong.
I received a personal invitation from Mike Taylor to contribute to this forum, so I will do so. Then I will also try to “get back to writing about science”. I promise.
First, let us get back to the main point of the initial post, whether now that the focus is on the individual paper, the journal format might become obsolete. We only included a couple of lines about this in our paper. We figured that it was another consequence of papers being individually available, but not directly related to the weakening relationship between IF and citation rates. Normally you do not want to stray too much in the discussion, or reviewers get upset. Actually, we arrived at a different prediction, not that journals will disappear, but rather that they will amalgamate.
Sorry, I am too lazy to paraphrase. We wrote “For the past few centuries journals were a convenient way to organize papers by subject, but search engines now allow us to find individual papers on specific topics from across the entire spectrum of journals, so highly subject-specific journals might become obsolete or begin to amalgamate”. But like this “Mike” character points out above, it depends on our reading habits. Time will tell.
I originally had another bit that was left out, about journals and issues being like file cabinets and folders (physical folders). Ten or 20 years ago we used to maintain this organization system in our computers, with files within folders within folders, organized by our personal and unique classification system. Now, with the development of better search engines, we can actually dump all our files into ONE folder. Sometimes we do not even have to tag the files, and the search engine will still find them relatively easily.
About the criticism of the paper, first, thinking of yourself as a reviewer is an unnecessary burden, more so for those of the feline persuasion. The paper is accepted/in press, so extensive new analyses are unrealistic, but new ideas are always welcome to help us in our future endeavours.
As Cameron Neylon indicates, we do not explain every dip and jump in the data. We had the hypothesis well before we got our act together and actually tested it. The point was not to explain all patterns in the data, but rather to test a hypothesis. The way I learned it was hypothesis –> prediction –> test –> support/reject/modify the hypothesis –> start again.
The data ARE available from Thomson Reuters, at a cost, so anyone is free to carry out similar or improved analyses. Off the top of my head I can think of several new hypotheses and types of analyses that could be carried out, but of course, it would be unwise to post my best ideas publicly.
The point is that the hypothesis is supported. The pattern is fairly evident. I was not aware of all the excitement on Twiter and the blogosphere, but I appreciate the attention.
Indeed, “these are uncertain times for publishers — and academics.”
Hi George, nice to see you here.
I wonder if you could respond here, briefly, to the specific points I brought up above? e.g., how did you decide, a priori, exactly where you would ‘cut’ the time series in Fig. 1, to fit the 3 different regression models? What is the statistical justification for this rather then a single regression model fitted to all the data? I didn’t find any information in your manuscript about this. It’s an enormous data set, impressively compiled, but
does not strike me as a robust, replicable method of analysing the data. As someone else asked above, do you have any idea why these “clear break[s]” arose? Could you ask someone to look at your Fig. 1 without the regression slopes and be confident that they would always see the same “clear break[s]”?
This is what troubles me a bit – it looks very much like you’ve arranged the data to fit your hypothesis, rather than the other way around. Please correct me if I’m wrong though.
Interesting post Stephen. Two thoughts on IF (from someone who hates the damn thing). First, even taking the arXiv paper at face value, IF is still doing a reasonable job of predicting citations (I suspect it would be hard to find a better single predictor). And related to this, from my reading of the Finch Committee notes it looks like we may be heading towards a system (in the UK) where Universities control a publication fund and thus have more input into what does and does not get published, with a greater emphasis on ‘quality over quality’. This really concerns me for a number of reasons, but pertinent here is, how will ‘quality’ be defined? Remember, this will be presubmission. Given the innate conservatism of university administrative structures, I suspect that the demise of the IF is a way off yet…
Now I should respond without descending into pedantic snark about a typo:
Apologies if this has already been mentioned as I have very limited time and I can’t read all the comments, but one aspect of the journal model that would need a good alternative replacement should journals disappear is the selection of peer reviewers. High impact journals don’t just attain their ‘quality’ from the articles published, they also attain it from the ‘quality’ of the peer reviewers they are able to recruit for their submissions. An excellent article has invariably had a thorough peer review from top people in the field and those top people have generally agreed to do the review because it is a top journal asking.
Given the overwhelming numbers of articles that some people are asked to peer review, the choice can come down to the name of the journal that is asking and, good or bad, if it is Nature or a no-name journal doing the asking, Nature is going to win out 99% of the time. In a journal-less world, what benchmark will people use to decide whether or not to do the reviewing?
Of course, the next question, “is it necessarily a bad thing that they don’t know?” is a whole other debate.
Pingback: Around the Web: Open access will bankrupt us, What data can and cannot do and more « Random Information
Pingback: Open Access: Money and Data talk and say the same thing? | One Change a Day
Pingback: Opening Science » Stephen Curry on the >Big Deal< article, posted below, in confrontation with other aspects and opinions concluding that: These are uncertain times for publishers — and academics. Change is coming, and perhaps more than any
Let us forget about this paper for now. Why would anyone still cling on to the IF as a valid measure of journal quality? There are many well known problems: (1) some types of publications within journals, such as letters and commentaries, are used to count citations (the nominator), but do not themselves count as “papers” (the denominator), and hence inflate the journal’s IF, (2) the IF depends on the number of references, which differs among disciplines and journals, (3) the inclusion of journals in the database depends solely on Thomson Reuters, a private company, and not on the fields’ practitioners, (4) the exact IF published by Thomson Reuters cannot be replicated using publicly available data, (5) the distribution of citations/paper is not normal, so at the very least the mode or median ought to be used instead of the mean, (6) the 2-year span for papers followed by one year for citations is completely arbitrary, and favours high-turnover over long-lasting contributions, and (7) journal editors can manipulate and artificially inflate their IFs.
So, is it the lack of a better measure? That might be an issue for publishers, but individual, INDEPENDENT, UNBIASED researchers should have abandoned the IF long ago.
Our analysis simply identifies one more problem: during the digital age the relationship between paper quality and IF is weakening, so the IF is losing its significance as a measure of journal quality.
Why was 1990 chosen as the year when the “digital age” started? We were there and we thought it was about right. What do you want instead? 1991? 1992? 1993? it does not make that much of a difference. I assume it is possible to do a similar analysis taking into account when each individual journal went digital. If we are correct, the result would be similar. Please go ahead and test it.
As to what happened before 1990, well, the hypothesis has nothing to do with that. The increase int he strength of the correlation between the IF and paper’s citations from the 1960 to 1990 probably has to do with the fact that the IF was first defined and started to be used in the early 1960s. Libraries used the IF to decide what journals to buy, and those journals were available, and their papers were cited. When papers started to be read independently of the journal, roughly around 1990, then everything changed.