“Ah! Then yours wasn't a really good school”
While I was at the University of North Carolina, I was invited to give seminars at various institutions. One of these was Princeton where I met John W. Tukey, an extremely able mathematician and statistician. In addition to his work at Princeton, John had an important job at Bell Labs where he practiced and encouraged the use of statistics. John and I respected one another, but we didn't always see eye to eye.
Some of my early research at ICI had concerned “tests of statistical significance.” That an effect is “statistically significant” means that it is unlikely to be a result of chance. You might, for example, be testing the efficacy of a new drug and you might want to check that the difference in efficacy between this and the standard drug was not just a result of experimental error.1 Clearly questions of statistical significance must be considered because without them the scientist can, on the one hand, be “chasing red herrings” or, on the other, be missing important small differences.
Now there are a variety of tests of significance to choose from and they all involve assumptions. In particular, we may assume that we know the form of the probability distribution of the data (the probability distribution of the “noise”). Some tests assume that this distribution is of a particular kind called the “normal” distribution and this is in fact a distribution that quite often can approximate reality. The difficulty ...