Experiment!: Planning, Implementing and Interpreting

8.10 Interpreting p-Values

It is common in many fields to report p-values from significance tests in scientific publications. Some statisticians are critical to this focus on the p-value because its meaning is frequently over- or misinterpreted. It is important to remember that the p-value is connected with the null hypothesis and nothing else. It measures the probability of obtaining data that are at least as extreme as the actual sample, under the assumption that the null hypothesis is true. Technically, this is not equivalent to the probability of the null hypothesis being true.

As explained in the example about Sven's fuel consumption, a significance test helps us to draw a more robust conclusion where a subjective judgment of the data could lead to ambiguities. Although the significance level is a somewhat arbitrary number, it is a quantitative means for motivating a conclusion. Nonetheless, the p-value is not entirely objective. Consider the t-test, for example. Here, the p-value is directly related to the observed t-value (a great tobs corresponds to a small p). Equation 8.1 shows that the magnitude of tobs depends on two things. It becomes large if the sample mean deviates substantially from the hypothesized population mean, that is, if there is a large effect in the data. But it also becomes large if the sample size n increases. In other words, it measures the effect in relation to the uncertainty in the data.

With a large sample size the uncertainty decreases and even ...