It’s always nice to read a paper that is obviously wrong, but where you have to think about why it is wrong. Because it makes you, well, think. And sometimes learn something new. So when I see a paper in TREE with the title “Do simple Models lead to generality in ecology?“, it’s clear that it’s going to answer “no”, and that I’m going to disagree.
In there paper, Evans and a plethora of co-author present this argument:
[W]e argue that there is usually a trade-off between simplicity and generality, such that simpler models are, all other things being equal, less general than are complex models. For example, a nonlinear population growth equation such as dN/dt = αN + βX1+a represents a large family of models, the members of which correspond to the constant parameters α and β being set to particular values (whereas a can take any value). If β is set to zero, we obtain a simpler linear equation, dN/dt = αN. Obviously, the nonlinear equation includes the linear one as a special case. Thus, the more complex equation represents a larger family of models than the linear ones and, therefore, is more general. It can pick out all the real systems that are described by the linear equation plus a range of others.
(I have slightly changed the equation from what the authors presented, to make my argument clearer. I think the original argument is unchanged, though)
i.e. simple models are less general because they cover less of the model space. Which is odd, because I think this is precisely why a simpler model is more general.
The argument for the greater generality of a simple model is that any mechanism not in the model is not relevant, so the conclusions we get from the model are relevant whatever other mechanisms might be acting. In other words, if we are using the simple model dN/dt = αN we are assuming other models like dN/dt = αN + βX1+a, dN/dt = αN + βeY, dN/dt = α sin θt N etc. etc. will show the same behaviour that we are investigating. Of course, they will differ in other ways but the purpose of any model should be to investigate specific phenomena, and how it behaves outside of that is generally not relevant.
On this reading, a more complicated model is less general because it fixes more of the model: dN/dt = αN + βX1+a precludes terms like eδX. Less abstractly, if we have built a model with sexual reproduction, we should not be using it to make inferences about asexual populations.
Of course, we can expand a model to have proportions of sexual and asexual reproduction, which would be more general than one that only assumed one mode of reproduction. But in practice, I think more complex models are usually developed in a different direction: they are made more complex by adding more mechanisms (e.g. age-structured reproduction, individual-based behaviours). These end up being much less general: a model of the behaviour of Cromerian giraffes will work well to predict their behaviours (such as cycling), but will not be of much use for predicting the head-perching behaviours of parrots. Luna, behaving on the top of a head
Evans et al. suggest that complex models can be made general by exploring their behaviour over a range of parameters:
The price for complexity is that such models usually need to be tied to data from specific systems. We are still left with the problem of how to generate general insights from models that are tied to specific systems. A key strategy currently used to solve this problem is the use of simulation experiments. Such experiments are performed on models, but parallel the kinds of experiment performed on laboratory systems. Techniques include analysing confounding factors, such as heterogeneities and stochasticity, changing the number of types of entity and process considered in the system, and systematically varying the parameters and variables of the model to determine whether its predictions are strongly or weakly influenced by changing values and, thus, which processes are more or less important dynamically. Utilising such simulation experiments requires the systematic consideration of possible (but not actually occurring) scenarios to understand the scope and limits of the model in question.
In tying a complex model to specific systems, a modeller is making assumptions specific for that system, which makes it less general (e.g that giraffes cannot fly, or unicycle beyond certain points). Exploring the (large) parameter space still only explores the model within these assumptions. In practice the lack of generality is even worse, because with more parameters it is more difficult to explore the parameter space: with a simple model like dN/dt = αN we can fully characterise its behaviour over all possible values of α. But with (say) 20 parameters and a model too complex for an analytic analysis, some choices have to be made about which parameters to explore, and what parameter values to use. We simply don’t have the computing power to explore the full space of parameters. Thus any generality that is gained by adding complexity is lessened simply because we are human, and cannot understand the model across the full range of its generality.
The other problem with using complex, specific, models to explore general behaviours is that it becomes difficult to understand why a model is behaving the way it is. Modellers are reduced to carrying out experiments with specific choices of parameters, and then using statistical methods like ANOVA to summarise the behaviour of the model. In essence it is treated as a black box. But the point of a model of general phenomena is to understand the systems where we see the phenomena: to unpack the black box. I suspect that the complexity of models is used as an alternative to thinking: a model is built to predict the behaviour of Cromerian giraffes, and subsequently the modeller thinks that the model can be used to explore general locomotory behaviours, without the bother of thinking about what aspects of locomotion (unicycling, flying, attraction to balding heads etc.) are the most relevant. Thinking first, and then developing a minimal model should be a preferable path, as it removes unnecessary parts of the model. this makes it easier to focus on what is important and thus easier to understand what the model is telling you. It also means there is a smaller parameter space to explore, and if you are simulating, each simulation will usually take less time (because there are fewer functions to evaluate). What’s not to like? Other than having to think, of course.
Prediction
One strange idea that crops up in this paper is this:
[Simple models] do not need to be tested against specific data because they represent concepts rather than systems.
and
Unlike in physics, where general models have to make testable predictions, ecology has embraced an approach where models claiming generality are untestable in any real system
Frankly, this idea is rubbish. Evans et al. criticize Volterra for comparing his predator-prey models (which produce cyclic dynamics) to data on fisheries in the Adriatic (where the data have cycles) for being confused. Likewise:
May and Anderson in 1979 compared the output of their model with data from real populations exposed to diseases. They found that ‘some of the theoretical conclusions can be pleasingly supported by hard data, while others remain more speculative’. This confusion of model purposes gives the false impression that simple demonstration models can provide actual explanations of specific systems.
But no explanation for the confusion is given – apparently it’s just that Volterra, Anderson, and May all erred by not doing things the Evans et al. way. Even if simple models “show that the modelled principles are sufficient to produce the phenomenon of interest” (which is a reasonable point of view), they are still intended as explanations of the real world. So surely they have to be tested against the real world, one way or another. To deny that is to deny that simple models of real-world phenomena are models of the real world. Idealised, yes, but they are still meant to say something about the real world.
I think the test of a simple, general, models could be rather informal. Volterra showed that predator-prey dynamics can produce cycles. One does not need to fit his model to the data to say that it works as an explanation: it is enough to see cycles in data with predators and prey. At Intecol last month, Sarah Calba gave a presentation about “prediction”, and argued that the concept is multi-faceted. We predict about the future (e.g. what the distribution of Cromerian giraffe will be in 2063), but we also make different sorts of predictions about the past and present, e.g that Cromerian giraffe cycle. And this can be tested, by going to Cromer and looking for giraffe, to see if any are cycling. The predictions are more qualitative, and serve a different purpose: they let us test our model explanations of the natural world, rather than tell us about the state of system that we are predicting for.
The beauty of simple models is that they help us understand some aspects of a range of real world systems. Volterra built his model to explain cycles in Adriatic fish. But the same model also provides an explanation of cycles in North American lynx. And chemical reactions. And it is also used in economics. But apparently it’s not general.
Reference
Evans MR, Grimm V, Johst K, Knuuttila T, de Langhe R, Lessells CM, Merz M, O’Malley MA, Orzack SH, Weisberg M, Wilkinson DJ, Wolkenhauer O, & Benton TG (2013). Do simple models lead to generality in ecology? Trends in ecology & evolution DOI: 10.1016/j.tree.2013.05.022
(Disclaimer – I’ve only skimmed through this and the paper, so I may have got the wrong end of the stick.)
This seems bizarre. I think the idea behind that paper is spot on – they seem to be arguing against a tendency to argue for simple models when a more complex model is both required and feasible (if difficult). I have a lot of time for that – there are some very good people in the author list.
The baffling thing for me is why “general” seems to have been confused with “flexible”. More complex models are (often) better because they can fit the data/problem/theory better. And that’s because they’re more flexible, not because they’re more general.
So a good adage (which they shy away from) is surely “simple means inflexible”?
I think your blog above Bob should actually go to TREE as a response paper!
Thanks for the interesting blog, Bob.
I think you somewhat misread what the paper is saying. A Volterra-type model demonstrates that a particular mechanism can produce a pattern that is broadly consistent with the data. But equally, a number of other mechanisms could too. So, if you have a range of different simple models that demonstrate mechanisms how do you choose between them? The answer surely lies in comparing them to data and asking of a particular system “does mechanism X underlie the observed?”. As the data are generated by a complex system (including the bugbears of age structure, stochasticity, spatial heterogeneities etc) it is difficult to imagine a way of generating a statistical fit to discriminate between mechansims that doesn’t incorporate some of the real world complexities of a specific system…and then you are into “complex models”.
My experience comes from decades of working with mites. I started that off as a “simple system” to understand the relationship between environmental noise and dynamics and to test a range of ideas about noise colour, extinction risk etc. No matter what modelling techniques, our ability to predict a system response to shocks (or in other words, understand it biologically) depends on matters like age structure and parental effects (as increasingly is shown in other well-studied systems). These factors determine the relationship between density and resources and their allocation, so they define the density dependent function. In the majority of systems that we are interested in, similar factors will play a part. Hence a simple model assuming density dependence is a simple function of N (or all individuals are dynamically equal) is more general in that it potentially could describe any system, but actually probably doesn’t correspond with any system. Understanding the outputs of such a model is akin to saying “if the world was like this, then this would happen…but we know its not, so it won’t”. In what sense is that generic understanding?
Conversely, if mites, drosophila, nematodes, rotifers, soay sheep, red deer, salmon etc etc show that age/stage structure, interacting with resource supply etc determines the density dependence and that models with such detail produce outputs that are qualitatively different from models without, then the generality may come from the knowledge of determining processes, coupled with analysis of the model outputs looking for similarities and differences.
The real motivation of the article is to encourage us, as a community, to think about these issues. The real world challenges of environmental change means that prediction is ever more important. Finding ways to build models that predict well is therefore a real challenge for us. If the referee community continues to say “complex models are less interesting because they can’t give us general insight” then the downside is that the world suffers. Simple models have a role, but it is not necessarily the case that simple=general=best.
Thanks for your response Tim, and sorry for the delay in replying.
I’m afraid I didn’t read the paper as saying what you’re saying (I guess that’s obvious): I certainly didn’t see it as a discussion about how to chose amongst several models (or perhaps amongst the mechanisms) are correct.
I think ecology has a much richer epistemology than you give it credit for. With simple models we usually as whether a model can explain the data, at least quantitatively. So Volterra’s model predicts that populations of predator and prey can cycle, and we don’t need another model predicting the same pattern before we can ask whether Volterra’s explanation is correct. We can see it working in a range of systems where we can clearly identify predators and prey. That’s enough to say it’s a reasonable explanation: what happens next is a different step: perhaps experiments are done with mites on oranges, or a more complex model is developed to look at model mis-fit for a specific case, or perhaps an alternative explanation is developed and then compared. But in practice scientific epistemology isn’t neat and simple.
I fully agree that if you’re trying to predict a specific system, then you’re almost certainly going to need a more complex model, but that’s because you’re asking specific and more detailed questions about the system, so the trade-off between simplicity and fit should push the model towards model fit, and simple=general=best is clearly wrong. But that’s because the general=best and simple=best parts are wrong, but that’s because of the type of problem. Framing the discussion around the simple=general part (as the title and abstract do: “We argue here that viewing simple models as the main way to achieve generality may be an obstacle to the progress of ecological research.”) seems an odd approach.
*cough*
*cough cough*
It’s obviously catching. Thanks for the correction.
Hi,
here is my take on this: http://www.petrkeil.com/?p=1567
Hi Bob, I’ve only had a quick read through the paper but my thoughts on this are much closer to yours than theirs (although I don’t completely buy your take). It seems to me that we don’t place enough emphasis on the role of prediction in defining ‘generality’. If model A is ‘more general’ than model B it implies, to me, that model A will make acceptable predictions (again, this needs to be defined) in more contexts than model B. So, in more places around the world, at more times in the history of the world, in more different kinds of habitats, for more taxa, at more spatial and temporal scales, etc. I suspect it is a reasonable general rule that the more variables we add to a model, the more interactions among variables we include, and the more complex the functional response among variables (including explicit modelling of co-variation among variables) the more likely it is to be idiosyncratic to the system/taxa/time period/scale from which the model was built. However, this is a reasonably simple empirical question – it’s surprising there have been few attempts to test it (although I haven’t either). That said, my experience with regression trees is that when we use ‘predictive ability for new data’ rather than ‘previous knowledge and logic’ as the stopping rule we almost always end up with much simpler trees. All the best. Jeff Houlahan
That sounds like a fun NCEAS project to run!