{"id":105,"date":"2008-08-19T18:26:41","date_gmt":"2008-08-19T16:26:41","guid":{"rendered":"http:\/\/occamstypewriter.org\/boboh\/2008\/08\/19\/why_p_values_are_evil\/"},"modified":"2008-08-19T18:26:41","modified_gmt":"2008-08-19T16:26:41","slug":"why_p_values_are_evil","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/boboh\/2008\/08\/19\/why_p_values_are_evil\/","title":{"rendered":"Why P-values are Evil"},"content":{"rendered":"

Over the last week or so I have been asked a couple of statistical questions where the root cause of the problems has been the unthinking application of p<\/em>-values. In both cases, the solution was to use confidence intervals instead; i.e. to look at the size of the effect, not whether it was different from zero. I decided it was time to take the battle to the host of p<\/em>-values, armed only with a t<\/em>-test. It is enough.
\nWhat is so bad about p<\/em>-values? Let’s start with a question: what does a p<\/em>-value tell you?<\/p>\n


\nIf you answered “the probability of getting the observed statistic, or something more extreme, if the null hypothesis was true”, then you’re right but you’re probably only parroting the textbook. If you said something about it giving the strength of evidence that a hypothesis is correct, then you’re wrong, but in good company: that is how most people use them in practice.
\nTime to spring an example on you. Let’s suppose we are interested in whether rocket scientists are more intelligent than brain surgeons. So, we round up a large number of both professions, and give them IQ tests. We then compare the distributions, to see which one has the higher mean. This is a classic t<\/em>-test problem. We get these results:<\/p>\n\n\n\n\n
Profession <\/th>\nMean <\/th>\nVariance <\/th>\nn <\/th>\n<\/tr>\n
Brain Surgeons <\/td>\n 110 <\/td>\n 6^2^ <\/td>\n 1000 <\/td>\n<\/tr>\n
Rocket Scientists <\/td>\n 111 <\/td>\n 6^2^ <\/td>\n 1000 <\/td>\n<\/tr>\n<\/table>\n

The difference is 1, with a standard error of 0.27, so the t-statistic is 3.7, with 998 degrees of freedom. Hence horribly significant: p<\/em>=2×10^-4^ we give it three stars, rejoice, and send off the paper.
\nThen our evil rivals in the physics department do the same study. Here are their results:
\n|_. Profession |_. Mean |_. Variance |_. n |
\n|Brain Surgeons | 112 | 6^2 | 50 |
\n|Rocket Scientists | 114 | 6^2 | 50 |
\nwhich gives a t<\/em> statistic of 1.67, and gives p<\/em>=0.1. Not significant, so no stars. They then publish a rebuttal calling us fools.
\nBut, look more closely. The variances are the same (you would never guess this is fictitious data, would you?). And in the second study the difference is 2, twice as large as the difference in the first. And yet it says there is no difference!
\nAh, some of you are saying. Look at the sample sizes. The second study is much smaller: the standard error is larger (by 20\/sqrt(2) times). We can see exactly what is going on if we look at the formula for the t-statistic, assume equal sample sizes in the two classes and re-arrange it:
\n\"\"<\/a>
\nwhat we see is that t<\/em> is proportional to the square root of n<\/em>.
\nSo what? Well, I would contend that for almost every significance test that is done in practice, the null hypothesis is, a priori<\/em>, false. For example, it is unlikely that the average IQs of brain surgeons and rocket scientists are exactly the same. It might be that the difference is only 10^-16^, but that is still not zero. So if we know that the hypothesis is false, why are we testing it? The answer is that we want to know if the difference is large enough to be important (in whatever sense). If brain surgeons’ IQs are 0.02 points higher on average, then the difference is not worth bothering about. But if they are 15 points higher, then they are more intelligent than rocket scientists.
\nSignificance tests, however, tell us something different. They tell us if the distribution is statistically significantly different from zero. If the test statistic is large enough, we declare the difference significant. But the test statistic depends on the sample size: with more data our estimates get more precise, so it is easier to distinguish a difference from zero. We saw that the t<\/em>-statistic increases with the square root of n<\/em>; other tests have the same general property1<\/a><\/sup>.
\nNow, if we know that the difference is not zero, then what does a significance test test? It depends in equal measure on (1) the true effect size (e.g. the difference between treatments), (2) the variability in observations (i.e. their standard deviation) and (3) the sample size. We know the first is non-zero: it is the signal we want to see. The variance tells us how difficult this is: the larger the variance, the more noise there is in the way. The sample size indicates how hard we are looking: a larger sample size means we have more information to try and see the signal. As we know the signal is there, the only thing a significance test can be testing is whether there is enough data to see the signal. In other words, it principally tells us about the quality of our study.
\nWhat is the solution? The first thing is to realise that statistics is no substitute for thinking. If you are a scientist, you should be able to decide for yourself what an important effect is, e.g. if a difference of 8 IQ points is important. Now, statistics can help here by providing the, um, statistics that are important. For example an effect size (e.g. a difference between control and treatment, of the slope of a regression line), or the amount of variation explained (i.e. R<\/em>2<\/sup> in regression). We can also tell you the uncertainty: that’s confidence intervals and standard errors. So if you estimate a difference of 0.2 IQ points with a standard error of 20, you know the estimate is uncertain, and there could still be a realistic difference. But if the standard error is 0.0002, the estimate is precise, but still small (remember that the population standard deviation is about 15<\/a>).
\nSounds easy, doesn’t it? The difficulties are in the statistical modelling (which is my job), and in understanding what is being measured, and how that relates to the real world. That shouldn’t be my job – I am merely a number pusher.<\/p>\n

1<\/sup> Except for a few pathological ones<\/p>\n","protected":false},"excerpt":{"rendered":"

Over the last week or so I have been asked a couple of statistical questions where the root cause of the problems has been the unthinking application of p-values. In both cases, the solution was to use confidence intervals instead; … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-105","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts\/105","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/comments?post=105"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/posts\/105\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/media?parent=105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/categories?post=105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/boboh\/wp-json\/wp\/v2\/tags?post=105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}