Yay, and I’m doing it without first having to break any laws. The official reason for this is to do some work: finish off writing some papers we started years ago (if we have the time, we might prod bulbous squidges with a sharp stick), and also perhaps start some new stuff that we won’t finish for another half decade. Of course, the real reason is to make GrrlScientist really jealous see a new country, and take photos of bits of it.

So, for the next couple of weeks I’ll mainly be in Townsville, which seems to be most of the way up the eastern side of Oz. I’ll also be in Cairns for a couple of days, to build another small pile of rocks.

The Original Cairn, which I will be nowhere near

I’ll endeavour to blog about my travels there (Day 36: First draft of Introduction completed, almost out of fingers. Hordes of budgerigars still at the windows, desperate to show us their new cycling tricks). So, ironically, I might actually do more blogging whilst I’m away: when I’m at home I’m constantly being distracted by hordes of parrots desperate to remove my fingers because I didn’t scratch their head correctly. And as for the rams…

Whilst I’m flying, your homework is to read up on why my beliefs are now protected in the UK workplace. If you don’t hear from me for 5 weeks, it probably means I decided to take the boat.

In it he lays out a view that is perhaps selfish, but understandably so. He outlines why he only agrees to review few papers, and what sort he will review:

For me to accept an invitation to review, a paper has to report novel and interesting results. If it has been circulated as a preprint on arXiv, then I don’t benefit from seeing it a second time as a reviewer. Similarly, the paper must also pique my interest in some way. Reviewing a paper that is reporting well-known facts (like documenting the growth of open access journals, for instance) is just plain boring. Test a new hypothesis, apply a new analytical technique to old data, or connect two disparate fields, and you’ll get my attention and my time.

The only other category of manuscripts that I’ll accept for review are those that are so biased or fatally flawed that it would be a disservice to the journal or to the community to allow them to be published. These papers must really have the potential to do harm (by distorting the literature or making a mockery of the journal) for me to review them.

Which; I guess, means he’s only interested in reviewing a small percentage of papers that come his way. As a journal editor, I find this attitude worrying, but as a potential reviewer, I understand it perfectly: there is only so much time in the day, so I don’t really want to spend it reading boring manuscripts (a plea: if you find yourself wanting to not review a manuscript, please suggest another victim, e.g. a post-doc or senior grad student. That really helps me, and whoever gets to review gets experience).

Davis’ solution to the problem is to suggest that a boring paper doesn’t need to be reviewed fully:

Perhaps all that is needed is to send null and confirmational results through perfunctory editorial review. These articles may only require passing a checklist of required elements before being published in a timely fashion. The result may be a cheaper and faster route to publication, and for some kinds of publications, this is exactly the desired outcome.

As an editor, my reaction to this is “eek”. In the comments, Kent Anderson lays out an important complication:

The purpose of peer review is both to improve the paper and to help it find the right outlet.
…
But let’s not kid ourselves — not every paper needs to be peer reviewed as rigorously as some, and there is no single thing called “peer review.”

Finding the right outlet is something that editors can certainly help with through a perfunctory review. At Methods in Ecology & Evolution, we receive quite a few papers that are not suited to us, and I try to suggest alternative outlets. But improving a paper is something that often needs more time: someone who knows the areas has to read a paper more carefully, and look for areas where it can be improved. As an editor, I often don’t have enough knowledge about these areas, which is why we ask reviewers.

But I am also sympathetic towards reviewers who agree to review a paper that turns out to be boring papers. The sorts of papers Davis is discussing are ones that are boring because they replicate other work, without adding much novel. So they are a priori identifiable as not terribly exciting, but worthy, and this should influence the choice of which journal the paper is sent to. In particular, somewhere like the Journal of Negative Results – Ecology & Evolutionary Biology might be a typical choice. But given that we already know such a paper is going to be difficult to find reviewers, and is also not going to have a huge impact, can we not try to make them more palatable for readers, without compromising on the reporting of the work? My feeling is soo 2008: Yes, we can.

Most papers we receive at JNR are written like standard papers: an introduction that lays out the problem and the literature, the methods describing a sanitised version of what was done, the results, and finally a discussion explaining why this work is so important that Nature should have accepted it at once. My feeling is that we could improve the readability of papers if we cut down the introduction and discussion massively. This is especially so when the paper is a replication: the intro could be pretty much “We repeat the seminarsemolina work of Gee & Grant, but in the Greater Rumple Horned Snorkack. Read their paper for why this is so interesting”. And the discussion could be similarly brief: “We found similar results, but with a stronger effect of Ewok urine. This might be because of the smaller size of the Greater Rumple Horned Snorkack.” There is probably no need to explain why this is important for global warming: if it was that important, either someone else will have already said it, or you would be publishing the work in a journal with a higher impact factor, like PLOS One.

Would this help? I guess if we told potential reviewers the word count, it might. It also might help to get more negative results published, by making the barrier to writing them up lower. Would losing the text harm science? The only harm I can see is that students writing up negative results might not learn to put their work in context, but one would hope that not all of thesis is like this, and if that was seen as a problem, extra discussion could be added to the thesis rather than the paper.

Is this something we should try at JNR-EEB? And for those of you not in ecology or evolutionary biology, does this sound familiar, or is the tedium of the discussion section one only we face?

Sure many will be pleased to hear that Statoil have been permitted to conduct a huge development in the North Sea http://t.co/27sWylcF

February 15, 2013 2:22 pm via TweetDeckReplyRetweetFavorite

@Protohedgehog

Jon Tennant

Now, oil companies (and that’s what Statoil is: it has nothing to do with statistics) are often vilified for being behind global warming, so it’s nice to see them trying to help solve the problem. I thought it was worth explaining how they’re going to do this: luckily last week I met an oil engineer who explained it all to me.

The basic problem is that the earth is absorbing more energy from the sun that it can radiate back out into space. Current approaches to dealing with this have been to try to reduce the amount absorbed, by reducing the amount of CO₂ in the atmosphere. But Statoil are trying a different approach: they want to reduce the amount of energy reaching the earth. This can be done simply by reducing the surface area of the Earth. They will do this by, essentially, deflating the planet. The new bore holes are a vital part of this plan: they will drill through the earth’s crust (which, of course, its thinner in the depths of the sea) and extract liquids from the rock. Removing this volume of liquid will obviously cause the earth to shrink in size, and the liquids themselves will be treated thermodynamically and be safely vented off into the atmosphere (and hence into space). According to their calculations, this will reduce the amount of radiation hitting the earth’s atmosphere enough to off-set the current amount of warming.

So there you go: thanks to the quiet and under-appreciated work of Statoil, Shell, BP, Exxon etc. we will finally be able to stop and reverse climate change. Pints of Guinness all round!

and we can see that there is a lot of variation, and possibly some long-term trends (e.g. recently popes seem to have survived longer). But is a long-lasting pope followed by one who keels over quickly? Well, in statistical terms that would mean that lengths of papal reigns would be negatively auto-correlated. And we have a nice tool to look at that sort of thing: it’s called the autocorrelation function, or ACF. Our hypothesis is, in technical language, that the lag one ACF is negative.

We can calculate that, and we find that it is 0.15. So, it is (a) small, and (b) in the wrong direction. No p-values needed. But this might be an effect of the longer-term trends in the data. We can remove this by fitting a suitable smooth curve (a spline, for those who want to know), and look at that:

The pink line is the fitted line, with the (approximate) 95% confidence interval.

We can see the long-term trends: from about 600AD to 1100AD was not a good period for popes. And, but for a blip around the 15th Century life seems to be improving. But what about the lag 1 acf? Well, that is 0.02, so basically zero, and also still in the wrong direction.

All in all, I think this disproves the notion that long reigns are followed by short ones. Except, it might be that this has changed over time. If we only look at the residuals for popes who started their papacy from after 1800 (i.e. the last 14 popes), we get an estimate of -0.09, which is at least in the right direction. Except that the approximate standard deviation is 0.29, so much larger. If we only look at the 9 popes who started poping after 1900, we get an acf of -0.62, with a standard error of 0.30. So we might just about have crept up to statistical significance (the z-statistic is -2.1, so less than -1.96 for a 5% significance), but (a) the sample size is small, (b) the significance is marginal, and the large-sample used may be way off, and (c) I’ve had to poke around a bit to get to something which might be marginally significant, so there is a certain amount of data dredging: looking for rubies in the rubbish and not stopping until I find one.

All in all, a pope will not spring eternal. Sorry.

From tables like this we calculate all sorts of statistics, to measure things like the amount of specialisation in the overall network. One of the innovations Konstans suggested was not to calculate the statistics directly from the raw numbers (i.e. the table above), but instead to recognise that the data are the result of a process, and it’s more important to estimate the statistics for the underlying, real, interactions that the data are just a realisation of.

Whilst we (i.e. Konstans) were doing this work, I was musing about the larger context, and realised that this explained something I did a few years ago, and some other work I’d seen, as well as some more work published in the greatest journal known to man. I think all this work shows one way that ecology is (or rather should be) maturing in the way it approaches how it summarises and interprets data.

Although this shift appears technical, what underlays it is a large epistemic change. The old way of doing things was to view the data as what you have, and calculate the statistics on them. The statistics are then just summaries of the data. In contrast, the new approach seeks to get at the processes underlying the data, by modelling the way the data are sampled from this process. So, the statistics are now a summary of the actual ecological process, filtered through the data that has been collected. So we have moved from summarising what we observe to what we think is going on in nature.

This all sounds fine in theory, but what does it mean in practice? Statistically, the new approach should be better because the sampling effort is accounted for, and there are more natural approaches to estimating the uncertainty in the statistics. But I also think the shift to explicitly estimating properties of the population should help us link the data to the theory.

An example of this is Konstans’ other innovation in our paper. The statistics he’s interested in are calculated at the population level, but they are obviously the results of individual behaviours. The shift to a more explicit model of the population makes it easier to write the model as the sum of individual effects. This then means we can ask about the effects of a change in the number of individuals on the network we are studying, which obviously means something different to a change in how individuals behave. So this shift helps us understand what the statistic is measuring, and how it is affected by the normal ecological processes we know and love.

The focus on the underlying processes should also help us develop ecological theory – the data will (hopefully!) show us interesting patterns that need explaining, and the methods for calculating the statistics give a framework for developing the models, which can be fitted back to the data.

Why isn’t everyone doing this? One reason is that the methods are only now being developed. Perhaps a more important one is that the methods have not been implemented in easy to use R packages (would anyone like to implement Konstans’ ideas as an R package…?), and packages like poilog and mvabund implement some other community ecology ideas. Another reason is probably inertia: ecologists aren’t used to thinking in these new ways, and so are using the tried and trusted methods that they are used to. Perhaps what this new approach needs is some success in showing that we genuinely get better results: we find out something new, or show something different that’s a better indication of what is really going on out there.

Today one of the Nobel committees got their revenge, giving the Physiology or Medicine prize to work on embryology. Ha, take that!

So, coming up this week: prizes for the Big Bang tomorrow, and chemical evolution on Wednesday. Perhaps Dawkins will get the literature prize on Thursday, just to really wind up the US politicians.

Who, then for the Peace Prize? Any suggestions?

The JIF is used as a measure of quality of science: it’s use to assess people and departments, not journals. And this can affect funding decisions, and hence people’s careers. If we are to use a numerical metric to make judgements about the quality of science being done, then we need to be sure that it is actually measuring quality. The complaint is that the JIF doesn’t do a good job of this.

But a strong argument can be made that we do need some sort of measure of quality. There are times when we can’t judge a person or a department by reading all of their papers: a few years ago I applied for a job for which there was over 600 applicants. Imagine trying to read 3 or 4 papers from each applicant to get an idea about how good they are.

Enter the altmetrics community. They are arguing that they can replace the JIF with better, alternative metrics. They are trying to develop these metrics using new sources of information that can be used to measure scientific worth: online (and hence easily available) sources like twitter and facebook (OK, and also Mendeley and Zotero, which make more sense).

Now, I have a few concerns about altmetrics: they seems to be concentrating on using data that is easily accessible and which can be accumulated quickly, which suggests that they are interested in work which is quickly recognised as important. Ironically, one of the criticisms of the JIF is that it only has 2 year window, so down-grades subjects (like the ones I work in) which have a longer attention span.

But I also have a deeper concern, and one I haven’t seen discussed. It’s a problem that, if it is not solved, utterly undermines the altmetrics programme. It’s that we have no concrete idea what it is they are trying to measure.

The problem is that we want our metrics to capture some essence of the influence/impact/importance that a paper has on science, and also on the wider world. But what do we mean by “influence”? It a very vague concept, so how can we operationalise the concept? The JIF at does this by assuming that influence = number of citations. This has some logic, although it limits the concept of influence a lot. It also assumes that all citations are equal, irrespective of the reason for citation or where the citing paper is published. But in reality these things probably matter: being cited in an important paper is worth more than in a crappy paper that nobody is going to read.

But what about comparing a paper that has been cited once in a Very Important Paper to one that has been cited three times in more trivial works. Which one is more important? In other words, how do we weight number of citations against importance of where they are cited to measure influence?

I can’t see how we can even start to do this if we don’t have any operational definition of influence. Without that, how can we know whether any weighting is correct? Sure, we can produce some summary statistics, but if we don’t even know what we’re trying to measure, how can we begin to assess if we’re measuring it well?

I’ve sketched the problem in the context of citations, but it gets even worse when we look at altmetrics. How do we compare tweets, Facebook likes, Mendeley uploads etc? Are 20 tweets the same as one Mendeley upload? Again, how can we tell if we can’t even explicate what we are measuring?

If someone can explain how to do this, then great. But I’m sceptical that it’s even possible: I can’t see how to start. And if it isn’t possible, then what’s the point of developing altmetrics? Shouldn’t we just ditch all metrics and get on with judging scientific output more qualitatively?

Unfortunately, that’s not a realistic option: as I pointed out above, with the amount of science being done, we have to use some form of numerical summary, i.e. some sort of metric. So we’re stuck with the JIF or other metrics, and we can’t even decide if they’re any good.

Bugger.

The lyrics are below the fold.

There’s Sb, As, Al, Se,
& H & O & N & Re,
& Ni, Nd, Np, Ge,
& Fe, Am, Ru, U,
Eu, Zr, Lu, V,
& Lsa & Os & At & Ra,
& Au & Pa & In & Ga,

& I & Th & Tm & Tl.

There’s Y Yb, Ac Rb,
& B, Gd, Nb, Ir,
& Sr & Si & Ag & Sm,
& Bi, Br, Li, Be, & Ba.

There’s Ho & He & Hf & Er,
& P & Fr & Fl & Tb,
& Mn & Hg, Mo, Mg,
Dy & Sc & Ce & Cs.
And Pb, Pr, and Pt, Pu,
Pd, Pm, K, Po,
& Ta, tc, Ti, te,

And Cd and Ca and Cr and Cm.

There’s S, Cf, & Fm, Bk,
& also Md, Es, No,
& Ar, Kr, Ne, Rn, Xe, Zn, & Rh,
& Cl, C, Co, Cu, W, Sn, & Na.

These are the only ones of which the news has come to Harvard,
And there may be many others, but they haven’t been discovered.