The Experimental Human Sciences has to Deal with Individual Differences: X-Phi as a Case Study

In the Experimental Human Sciences, there is a tendency to equate statistically significant differences between vignettes or conditions or wordings as being indicative of gaining insights into people’s intuitions. This is based on a number of large assumptions, the primary one being the mean of a group is an adequate estimator for the intuitions of those in the group. While this is justified when interpreting the causal effect of different experimental manipulations on means of participants’ judgments, it is questionable about whether this is justified when extrapolating to what “people believe”. The problem is individual differences. Individual differences are the inescapable fact that: for all human traits, behaviors, mental states, abilities, characteristics, intuitions: people differ. Not everyone shares the same intuitions. Not everyone holds the same values. Individual Differences are present in every experimental study in the human sciences. They are, however, condemned to the denominator of our equations, with larger individual differences masking our between-group estimates and shrinking our effect sizes. Experimentation can highlight what factors exert a causal effect on intuitions and reasoning, but individual differences prevent us from making the leap to what “people believe”. In studies with large and robust experimental effects, there are individuals who understand our experimental paradigms yet disagree with the group. These are people too. We highlight examples of the confusion from experimental mean group differences are interpreted as what “people believe”. Using Experimental Philosophy as a case study, we further discuss that while Experimental Philosophy tends to have larger effect sizes than other disciplines, group distributions due to experimental manipulations in Experimental Philosophy still show over large overlap between groups. These problems are not unique to Experimental Philosophy, as it is indeed shared by all experimental human sciences. Unlike many other fields, however, Experimental Philosophy is interested in understandings people’s intuitions. We discuss how this problem can be addressed in future Experimental Philosophy studies by incorporating within-subjects experiments opposed to between-subjects experiments, highlighting these solutions using data from a replicated X-Phi study.