A frequent misconception is that the
p-value is the probability that findings are due to chance (alone) or – put in other words but generally meaning the same – the chance that the null hypothesis (
H
0) is true [
2]. Let us illustrate this misconception with a simple example. Suppose, you bought one ticket for a lottery for which one million tickets have been sold. Assuming a fair lottery (
H
0), every ticket has the same chance of winning and thus your chance of winning,
p, is 1 in a million (provided, of course, that the winning ticket is among those sold). This
p-value, for any single ticket, is a conditional probability – the probability of any single ticket winning the lottery given the assumption (the condition) that
H
0 (fair lottery) is true:
p(winning |
H
0) = 0.000001. Of course, one of the tickets must win the lottery. So, after the lottery is drawn, you now have the winning ticket, which
a priori had a chance of winning of 1 in a million. However, this does not imply that you can now determine that the probability that the lottery was fair given that you have the winning ticket,
p(
H
0 | winning), is 0.000001; the mere fact that your ticket was drawn contains no information about the fairness of the lottery (how it was conducted, who made the draw, etc.). Indeed, if we have assumed the null hypothesis that the lottery was completely fair, then the probability that your ticket was drawn by chance must be 1 (100 %). Null hypothesis significance testing always assumes random sampling, and under that assumption, in our example, the probability that findings are due to chance is always 1 (100 %). Take for instance a baseline difference between randomized groups in an experiment: this difference is entirely the result of chance, regardless of the
p-value of a statistical test on this difference.