{"id":277,"date":"2009-06-22T15:34:57","date_gmt":"2009-06-22T15:34:57","guid":{"rendered":"http:\/\/occamstypewriter.org\/rpg\/2009\/06\/22\/on_article_level_metrics_and_other_animals\/"},"modified":"2009-06-22T15:34:57","modified_gmt":"2009-06-22T15:34:57","slug":"on_article_level_metrics_and_other_animals","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/rpg\/2009\/06\/22\/on_article_level_metrics_and_other_animals\/","title":{"rendered":"On article-level metrics and other animals"},"content":{"rendered":"

What I wanted to talk about yesterday, before I got distracted<\/a>, was a mix of things that came out of last Monday’s ALPSP meeting, a paper in PLoS One and this whole assessment schtick that we’ve got going on.<\/p>\n

\nWe know, don’t we, that the Impact Factor<\/a> as a measure of an individual scientist’s productivity<\/em> is deeply flawed<\/a>. Some might go as far as to say it’s flawed fundamentally and should be ditched forthwith: after all it is prone to gaming, unduly skewed by review articles, does not demonstrate a link between the ‘quality’ of a journal and any given article therein, can reduce<\/em> the measured effectiveness of a single article (if a seminal paper is cited in a review then the review<\/em> rather than the original paper can be cited more frequently), has no meaning in fields (humanities, nursing\u2026) where citations are not routinely used as a measure of impact, and is very, very slow. (Seriously. We’re looking at a two year gap at least between the original work being done and it being cited in any meaningful manner\u2014there has to be the first paper published, experiments thought about and performed, results analysed, papers written, reviewed and finally another publishing round before you can measure it.) It is at least one step removed from the output it might be taken as measuring, relies on everyone else playing fair, and, just like a quantum physical experiment, perturbs in the act of measurement the very thing it is supposed to measure.<\/p>\n

\nThere have been attempts to massage the Impact Factor, and new formulations cooked up, such as the Hirsch Index<\/a> (and its variants) or the Eigenfactor<\/a>, but essentially these are all measuring not a researcher’s activity, nor even how good that activity has been, but rather how someone else has managed to exploit it. Let’s not even think about research activities that do not result in a classical ‘citable unit’: database annotations, talks, training of students, fixing kit<\/a>, etc., etc. And when you start looking at this in terms of stakeholders, for example the Research Excrement<\/del> Excellence Framework (&c.), you get into all sorts of difficulties because people want to be able to measure you in a pretty automated way in order to finish it before the heat death of the universe. <\/p>\n

\nClearly there is no substitute for actually reading an article to determine its importance<\/code><\/p>\n

\n\u2026 which is probably why certain people last Monday were very careful to say “indicators” rather than “metrics”, and brings me onto article-level metrics<\/a> and a paper on the impact of Wellcome-funded research.<\/p>\n

\nPLoS ONE are trying to look at the impact (however you define that<\/em>) in terms of individual articles. According to Peter Binfield<\/a>, for each PLoS ONE article the aim <\/p>\n

\nis to provide information relating to online usage, citation activity, blog and media coverage, commenting activity, social bookmarking, "star ratings" and "best of" picks as selected by academic experts, as well as other measures yet to be determined<\/code><\/p>\n

\nIn other words, you’ll be measured by what you write, not by who writes about what you wrote. And even that’s a bit tricky, because really, there is no substitute for “expert” review, as was elegantly demonstrated in PLoS One last week<\/a>.<\/p>\n

\nLiz Allen and colleagues at the Wellcome Trust analysed the output of a whole bunch of Wellcome-funded researchers, whether project or programme grant or fellowship holders. They appointed a panel, or college, of ‘experts’ to review papers published in 2005 and essentially got this college to answer the question “how good is this stuff, I mean, really?”\u2014for nearly 700 papers. (Okay, they didn’t assess ‘non-citable units’, but it’s a start.)<\/p>\n

\nThe results are interesting. For a start, it turns out that the Impact Factor or citational analysis is a pretty poor indicator of an individually paper’s importance (or quality, or influence). We all kind of suspected this, but it’s nice to finally have some data. <\/p>\n

\nThe other thing, and here I declare my conflict of interest, is that the Wellcome’s expert college on the whole matched Faculty of 1000’s<\/a> assessment. Yes, some papers were completely missed, but we think we know why and are working to make sure that in future we do spot more interesting things that appear in less well-known journals. We’re pretty stoked about this, actually.<\/p>\n

\nYou can’t beat human input. Not yet, anyway. What we’d really like to do is find some way of matching PLoS’s article-level metrics<\/del> indicators with our F1000 Factor and synthesize a sensible measure that looks at the quality of the research as well as its influence and usage. Here would be a good place to draw your attention to Johan Bollen’s incredibly cool network graphs<\/a>, which could provide hours of fun for all the family. And yes, I talked to Johan on Monday and yes, he does plan to slice those data at the individual author\/research group level.<\/p>\n

\nYou may have noticed that we (Faculty of 1000) already ‘score’ articles. Seeing as we have these scores, you might imagine that we’d be able to, oh I don’t know, give scores to individual journals or institutions: perhaps even individual scientists. Now there’s an idea\u2026<\/p>\n

\nBut you know, I do wonder if all these indicators and claims of unfairness, fairness, whatever, might be missing the point, somewhat. Whatever the metric you use, good science will out<\/strong>. I wonder if there are truly any brilliant scientists who have not got jobs, or not got funded, because the current indicators have missed them? And I’m sure you’ve all got stories you can tell me. To which I say,<\/p>\n

\nthe plural of anecdote is not ‘data’<\/em>\n<\/p><\/blockquote>\n

\nFurthermore, we all assume that there is necessarily a link between past results and future performance, in opposition to what the investment adverts keep telling us. The Impact Factor\u2014or any citation metric\u2014is at least two years out of date. So I wonder if this holds when dishing out the cash for research? Now, I’m pretty open-minded, and a strong believer in experiment. I would really like someone to test my hypotheses. So I propose the following:<\/p>\n

\nTake a pot of money. Tens, if not hundreds, of millions of pounds or dollars, doesn’t matter which. Divide it into four equal portions. Invite researchers to apply for the money in the usual way, and on receipt of the applications assign them randomly to one of four groups, A, B, C or D. Group A is judged in the standard manner of competitive grant applications. Group B is judged on the basis of Hirsch Indices or Impact Factors of the PIs involved and Group C uses something like an expert college in much the same way as the papers in the Allen study were judged (but looking at individual researchers rather than their papers). Applicants in Group D get the money randomly. We’d probably want to limit to a 20% success rate for each group\u2014someone else can work out the details.<\/p>\n

\nAnd then, after five years, we assess, using every best method available, the productivity of the researchers who were funded (and could probably look at the careers of those who weren’t, too). <\/p>\n

\nThis is probably unethical, but potentially very interesting. I wouldn’t be surprised if there were a few surprises.<\/p>\n


\n\"ResearchBlogging.org\"<\/a><\/span><\/p>\n

\nAllen, L., Jones, C., Dolby, K., Lynn, D., & Walport, M. (2009). Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs PLoS ONE, 4<\/span> (6) DOI: 10.1371\/journal.pone.0005910<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

What I wanted to talk about yesterday, before I got distracted, was a mix of things that came out of last Monday’s ALPSP meeting, a paper in PLoS One and this whole assessment schtick that we’ve got going on. We … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-277","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/posts\/277","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/comments?post=277"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/posts\/277\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/media?parent=277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/categories?post=277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/rpg\/wp-json\/wp\/v2\/tags?post=277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}