The WEIRD instrument problem and systematic bias in psychological research

As researchers broaden samples to populations that are not Western, Educated, Industrialized, Rich, and Democratic (WEIRD), they often use instruments designed for WEIRD populations. In this paper, I argue that this practice creates systematic biases that, for example, can explain consistent findings of less complex personality structures found outside WEIRD populations. First, I model how administering instruments designed for one, typically WEIRD, reference population to novel populations systematically underestimates the relative complexity of novel populations. This occurs because instruments optimized to measure the WEIRD population fail to capture the full structure of other populations. An additional outcome of the model was that a population's cultural distance from the reference population was strongly negatively associated with the measured personality structure complexity, even if the actual complexity for all populations was the same. I conducted an empirical analysis and found a negative association between cultural distance and personality structure complexity in pre-existing data. Consistent with the model’s prediction, cultural distance from the United States (the reference country for the personality instrument) had the strongest negative relationship with personality structure complexity of all 67 countries in the data. This was strong support for the hypothesis that lower apparent complexity outside WEIRD populations results from the use of WEIRD instruments, rather than from real differences in complexity. Modeling suggests that bias should occur whenever an instrument designed for one group, such as one gender or one cohort, is administered more widely. I propose possible reforms to avoid this type of bias.