The pragmatics of statistical inference

Psychologists often take statistical hypothesis testing to be a procedure that detects whether effects do or do not exist; the output of this procedure can then be used to inform theories, which should predict the effects that do exist, and not those effects that do not. Call this the modular view of statistics: Statistics can do their job relatively separately from the theories that are being tested, just so long as the right effects are being tested. The interface between statistics and science can be called the pragmatics of statistical inference. The modular view declares that interface is simple for hypothesis testing: It consists of science telling statistics what to test for; and statistics telling science whether it exists or not. One reason why there has been a credibility crisis may be that the modular view is dominant yet unhelpful. If statistics cannot in an atheoretical way declare what does and does not exist, then psychologists have not been genuinely testing their theories. In this chapter I reject the modular view. One might reject the whole concept of getting evidence of whether or not there is an effect; instead, I accept this aspect of current practice but argue one can only get evidence for a relevant effect being there or not given a theory of what size effect could be expected on that theory.