Another way to measure your research impact

The h-index attempts to reduce a researcher’s output to a single number: your h-index is the number of papers you’ve published, N, that have been cited at least N times. It seems like a broader measure than pure citation counts but is by no means a perfect measure. It is seen as mostly confirming past successes and it is variable between different subjects. Much has been written about it since it was first devised in 2005, and various attempts have been made to improve it.

A paper in this week’s Nature announces a new variant – the future h-index. This is designed to predict what your h-index will be a few years in the future, taking into account some additional factors. It seems interesting but still flawed.

Appearing in such a high-profile venue as Nature has given this new metric some prominence, and there has been much comment already (e.g. see this piece in The Scientist). On Twitter Noah Fierer commented:

In case you were wondering – secret to high H-index = lots of papers in high profile journals.

Hardly a surprising finding. The Chronicle of Higher Education has some further thoughtful comments on the new tool.

Konrad Kording, one of the authors of the paper in Nature, said that their future h-index has “proved more than twice as accurate as the h-index for predicting the future success of researchers in the life sciences”.

But Jorge Hirsch, the inventor of the original h-index is not impressed. The Chronicle reports

he said the factors added to his h-index appeared to have little meaningful effect. He suggested the additional factors had been devised by “optimizing the coefficients” for a particular set of authors covered by the paper. He said the predictive powers would not hold up for a wider set of test cases.

That echoes my thought.  Publication patterns and citation practices vary between fields, so basing a general formula on researchers in one particular field is not realistic. The article mentions factors such as the quality of training and the standing of one’s PhD adviser (and I would add one’s postdoc supervisor and later mentors) but none of these factors are included in the new index as they apparently have only a small effect.

But, hey, everyone knows that these magic numbers are basically just that – a data reduction too far. As Stephen Curry said on Twitter:

scientists invent a new way to screw themselves over

So I think Wired magazine has the best idea:

while neither one’s h-index nor the predictions of this equation are destiny, playing with this formula certainly is fun.

You can try the formula for yourself.

About Frank Norman

I am a retired librarian. I spent 40 years working in biomedical research libraries.
This entry was posted in Bibliometrics etc. Bookmark the permalink.

4 Responses to Another way to measure your research impact

  1. AJ Cann says:

    Inferential statistic fail 🙁

  2. Laurence Cox says:

    I have just been reading Lee Smolin’s book “The Trouble with Physics” and towards the end (Chapters 17-19) where he gets away from string theory and starts to discuss the structure of how we do science in general, he does make (I think) the good point that the approach we use now to rate the output of individuals (and by extension, departments) tends to favour what he calls “craftsmen” over “seers”. That is to say, by working in a popular field and making incremental advances one has a better chance of a life-long career in academia than if one strikes out into a new field (or one that was popular a long time in the past) and tries to solve a fundamental problem. The only example of the latter I can think of recently where someone succeeded was Andrew Wiles’ solution of Fermat’s Last Theorem where he worked on his own for over ten years. Do the OT bloggers think there are enough “seers” around and, if not, how do we change the rating method to get a better balance?

  3. Frank says:

    Laurence – that is an interesting categorisation. It reminds me a little of the dichotomy between ‘basic’ and ‘applied’ research. We need new ideas but we also need to develop and exploit those ideas. I wonder what is the ideal ratio of ‘craftsmen’ to ‘seers’?

    How can you measure where a scientist falls along the craftsman/seer axis? Perhaps there is a characteristic citation pattern for seers?

  4. Mr. Gunn says:

    If all you’re doing is counting citations, you’re bound to fail. They’re lagging noisy indicators. How many citations does a paper have to accumulate before it can be said to be significantly above baseline impact? It’s clear that a paper with 1000 citations over 5 years is more impactful than one with 5 citations over the same time, if it’s in the same field, but what about one paper with 30 citations and one with 40, again from the same field? Look how many years it takes for this clearly impactful subject to accumulate a significant number of citations:

    And that’s summing across all the papers on the topic!

    We have to find better, faster, metrics. Things that take into account actual usage of the results in some fashion, and I’m not saying all work has to be judged by how readily applicable it is. We want the Prusiners and McClintocks, we just want them to get better visibility faster. Part of this requires scientists to get over the reluctance to learn how to communicate effectively, and part of it requires dismantling this stupid nonsense we do where you have to have a CNS paper to be considered good. That’s even worse than counting citations, and it’s directly the cause of “seers” not getting the attention they deserve.

    Activity and usage data should be open, readily accessible, and each community of researchers, publishers, librarians, funders, patients, or anyone else, should be able to define the metrics that are meaningful to them, for their particular application at that moment. Any ranking system can be gamed, but you can’t game all the dimensions of impact all at once, so moving away from one universal metric would be one of the best things we could do.

    Not to be too critical, so very glad they’re thinking along these lines, just want to see a little more getting with the times.

Comments are closed.