{"id":373,"date":"2012-03-23T11:36:59","date_gmt":"2012-03-23T09:36:59","guid":{"rendered":"http:\/\/occamstypewriter.org\/boboh\/2012\/03\/23\/the_problems_of_interpreting_data\/"},"modified":"2014-04-09T15:14:54","modified_gmt":"2014-04-09T13:14:54","slug":"the_problems_of_interpreting_data","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/boboh\/2012\/03\/23\/the_problems_of_interpreting_data\/","title":{"rendered":"The Problems of Interpreting Data"},"content":{"rendered":"

A few weeks ago Science<\/i> published a paper<\/a> which claimed that biodiversity was important for the functioning of dry grasslands. This claim was strange because the analysis suggested that biodiversity wasn’t very important – it only explained about 4% of the variation in the data, whilst the abiotic components in the model explained about 50%. This caused me to look a bit deeper at the paper, and also to think a bit more about one particular aspect of the authors’ argument: how we can infer processes from observational data (i.e. where we don’t manipulate the conditions).<\/p>\n


\nIn their paper, Maestre et al<\/i>. argue that the effects they see will be even stronger than estimated:<\/p>\n

“[b]ecause we did not experimentally control for other abiotic and biotic factors that are known to affect ecosystem functioning, significant relationships would indicate potentially strong effects of richness on multifunctionality”<\/p><\/blockquote>\n

Which is utter rubbish. As with any good scientific discussion, the response can be summarised with an xkcd cartoon:
\n\"Correlation\"<\/a>
\n(check the roll-over text too)
\nBecause the authors didn’t “experimentally control for other abiotic and biotic factors that are known to affect ecosystem functioning”, any significant relationships could be spurious. Denying this would mean having to accept the stork theory of reproduction<\/a>.
\nBut there is a kernel of truth in the assertion; the authors were unfortunate in choosing a horribly wrong justification for a more reasonable statement. Because if there is an effect of species richness, then the authors are correct to suggest they are under-estimating the effect. And I suspect underestimation like this is rife in ecology.<\/p>\n

The Problem: errors in variables<\/h2>\n

I can illustrate the problem simply enough. Imagine we have a regression problem where this is our data:
\n\"A<\/a>
\nNow imagine that rather than having the actual covariate, we only have a corrupted version. Here I have added some random noise (the response is the same):
\n\"A<\/a>
\nThe corrupted data is more spread out, and the regression line for the corrupted data (the blue line) has a shallower slope than the regression for the original data. This is despite the correlation between the original and corrupted covariate being 0.9 – the pure and corrupted variables are pretty similar.
\nOf course that’s only one regression. But the pattern is general: when you have an error in your variable (i.e. what you measure is only correlated with what’s truly causal), the regression coefficient is biased towards zero. We can get an idea about how much by doing lots of simulations:<\/p>\n

\"The<\/a><\/p>\n

The blue line is the true regression coefficient. Even with a strong correlation there is a significant bias: the estimated slope is half if the correlation is about 0.7 (which is still pretty strong). With a correlation of 0.9, the regression coefficient is about 80% of the correct value.<\/p>\n

Does this screw up field ecology?<\/h2>\n

Back to the paper. The authors only estimated species richness, so if species richness were genuinely driving ecosystem function, there almost certainly will be some effect of the error in the variable. But even if it is estimated perfectly, I think the same problem is still present because I doubt that species richness per se<\/i> is what would affect productivity: the frequencies of the different species will also matter (imagine an ecosystem dominated by one species, with 19 other species which are all very rare – I think this would behave more like a single species ecosystem than one with 20 species). Even if biodiversity does affect ecosystem function, species richness is not the driver: that is will be more complex function of richness and relative abundance. But species richness is probably correlated with this function, so the “true” effect of biodiversity is underestimated.
\nMy hunch is that this problem is pervasive in ecology. We measure all sorts of things and see if they have an effect on nature, for example if water bird populations are affected by mean winter temperature. But mean winter temperature itself is probably not the driver – it could, for example, be the effect temperature has on freezing ponds and lakes. But the precise mechanism would be difficult to get at – is it the date each pond freezes over? Or freezes over enough that not more than 3 ducks can sit in the water? Or is it the length of time it’s frozen over or…<\/p>\n

You see my point – the mechanisms are complicated, and we only measure surrogates. Indeed, in most cases we can only measure surrogates. The depressing conclusion, then, is that we are underestimating the effects we are looking at, because even if we have found the correct ones we usually can’t estimate the accurately. So our models are wrong (although they may still be useful).<\/p>\n

We might not have to worry too much if the relationship between what we measure and the true driver remains constant. Then the bias is stable, and although our inferences are biased at least the bias doesn’t change. Our predictions will still be valid, albeit not as good as they could be. If the relationship does change then we could be screwed, though.<\/p>\n

All of this just makes interpretations more messy. If we estimate an effect of species richness (say), it could be that the true effect of biodiversity is larger and we are only measuring an indicator, or it could be that there is no effect, but both ecosystem function and species richness are affected by the same abiotic drivers.<\/p>\n

I wonder what you, my reader, thinks? Is this something to be seriously concerned about? I think the problem needs to be assessed from a wider perspective than I can give.<\/p>\n

Help?<\/p>\n","protected":false},"excerpt":{"rendered":"

A few weeks ago Science published a paper which claimed that biodiversity was important for the functioning of dry grasslands. This claim was strange because the analysis suggested that biodiversity wasn’t very important – it only explained about 4% of … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":32,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27],"tags":[],"class_list":["post-373","post","type-post","status-publish","format-standard","hentry","category-statistics"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts\/373","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/comments?post=373"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts\/373\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/media?parent=373"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/categories?post=373"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/tags?post=373"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}