Mindless statistics
Section snippets
The null ritual
Textbooks and curricula in psychology almost never teach the statistical toolbox, which contains tools such as descriptive statistics, Tukey's exploratory methods, Bayesian statistics, Neyman–Pearson decision theory and Wald's sequential analysis. Knowing the contents of a toolbox, of course, requires statistical thinking, that is, the art of choosing a proper tool for a given problem. Instead, one single procedure that I call the “null ritual” tends to be featured in texts and practiced by
What Fisher and Neyman–Pearson actually proposed
In discussions about the pros and cons of significance testing in the social sciences, it is commonly overlooked (by both sides) that the ritual is not even part of statistics proper. So let us see what Fisher and Neyman–Pearson actually proposed. The logic of Fisher's (1955, 1956) null hypothesis testing can be summarized in three steps:
Feelings of guilt
Let me introduce Dr. Publish-Perish. He is the average researcher, a devoted consumer of statistical packages. His superego tells him that he ought to set the level of significance before an experiment is performed. A level of 1% would be impressive, wouldn’t it? Yes, but … He fears that the p-value calculated from the data could turn out slightly higher. What if it were 1.1%? Then he would have to report a nonsignificant result. He does not want to take that risk. How about setting the level
Collective illusions
Rituals call for cognitive illusions. Their function is to make the final product, a significant result, appear highly informative, and thereby justify the ritual. Try to answer the following question (Oakes, 1986, Haller and Krauss, 2002):
Suppose you have a treatment that you suspect may alter performance on a certain task. You compare the means of your control and experimental groups (say 20 subjects in each sample). Further, suppose you use a simple independent means t-test and your result
An editor with guts
Everyone seems to have an answer to this question: Who is to blame for the null ritual? Always someone else. A smart graduate student told me that he did not want problems with his thesis advisor. When he finally got his Ph.D. and a post-doc, his concern was to get a real job. Soon he was an assistant professor at a respected university, but he still felt he could not afford statistical thinking because he needed to publish quickly to get tenure. The editors required the ritual, he apologized,
The superego, the ego, and the id
Why do intelligent people engage in statistical rituals rather than in statistical thinking? Every person of average intelligence can understand that p(D|H) is not the same as p(H|D). That this insight fades away when it comes to hypothesis testing suggests that the cause is not intellectual but social and emotional. Here is a hypothesis (Acree, 1978, Gigerenzer, 1993): The conflict between statisticians, both suppressed by and inherent in the textbooks, has become internalized in the minds of
Meehl's conjecture
Paul Meehl, a brilliant clinical psychologist with a broad interest in the philosophy of science, was one of those who blamed Fisher for the decline of statistical thinking in psychology. “Sir Ronald has befuddled us, mesmerized us, and led us down the primrose path. I believe the almost universal reliance on merely refuting the null hypothesis … is a terrible mistake, is basically unsound, poor scientific strategy, and one of the worst things that ever happened in the history of psychology” (
Feynman's conjecture
The routine reliance on the null ritual discourages not only statistical thinking but also theoretical thinking. One does not need to specify one's hypothesis, nor any challenging alternative hypothesis. There is no premium on “bold” hypotheses, in the sense of Karl Popper or Bayesian model comparison (MacKay, 1995). In many experimental papers in social and cognitive psychology, there is no theory in shooting distance, but only surrogates such as redescription of the results (Gigerenzer, 2000,
The dawn of statistical thinking
Rituals seem to be indispensable for the self-definition of social groups and for transitions in life, and there is nothing wrong with them. However, they should be the subject rather than the procedure of social sciences. Elements of social rituals include (i) the repetition of the same action, (ii) a focus on special numbers or colors, (iii) fears about serious sanctions for rule violations, and (iv) wishful thinking and delusions that virtually eliminate critical thinking (Dulaney and Fiske,
References (64)
The fallacy of the null hypothesis in soft psychology
Applied and Preventive Psychology
(2004)- Acree, M.C., 1978. Theories of Statistical Inference in Psychological Research: A Historicocritical Study....
- American Psychological Association, 1974. Publication Manual, 2nd ed., 3rd ed., 1983; 4th ed., 1994; 5th ed., 2001....
Differential psychology
(1958)- et al.
Null hypothesis testing: problems, prevalence, and an alternative
Journal of Wildlife Management
(2000) The test of significance in psychological research
Psychological Bulletin
(1966)Précis of “Statistical significance: rationale, validity, and utility”
Behavioral and Brain Sciences
(1998)The statistical power of abnormal-social psychological research: a review
Journal of Abnormal and Social Psychology
(1962)Statistical power analysis for the behavioral sciences
(1988)- Czerlinski, J., Gigerenzer, G., Goldstein, D.G., 1999. How good are simple heuristics? In: Gigerenzer, G., Todd, P.M.,...
Cultural rituals and obsessive-compulsive disorder: is there a common psychological mechanism?
Ethos
Significance tests die hard
Theory and Psychology
Statistical Analysis in Psychology and Education
The Meaning of it All: Thoughts of a Citizen-Scientist
The fifth edition of the APA Publication Manual: why its statistics recommendations are so controversial
Educational and Psychological Measurement
Statistical methods and scientific induction
Journal of the Royal Statistical Society (B)
Statistical Methods and Scientific Inference
Editorial comment
Memory and Cognition
Psychology and Life
Probabilistic thinking and the fight against subjectivity
The superego, the ego, and the id in statistical reasoning
Adaptive Thinking: Rationality in the Real World
The null ritual: What you always wanted to know about null hypothesis testing but were afraid to ask
Cognition as Intuitive Statistics
The Empire of Chance. How Probability Changed Science and Every Day Life
Hindsight bias: a by-product of knowledge updating?
Journal of Experimental Psychology: Learning, Memory, and Cognition
Cited by (622)
Statistical approach for radioactivity detection: A brief review
2024, Journal of Environmental RadioactivityWhen expert predictions fail
2024, Trends in Cognitive SciencesThe roles, challenges, and merits of the p value
2023, PatternsWhen perceived informativity is not enough. How teachers perceive and interpret statistical results of educational research
2023, Teaching and Teacher Education