Noise in the process: Assessing the evidential value of mediation analyses in marketing and consumer psychology journals

Mediation analysis plays a central role in marketing research and consumer psychology by modeling the complex processes underlying total effects. Like other inferential methods, tests of indirect effects are susceptible to false-positive results when multiple defensible analytic decisions can lead to different conclusions (i.e. researcher degrees of freedom). To assess the collective evidential value of mediation analyses in marketing, we (1) measure the relative proximity (RP) of confidence intervals to zero and (2) aggregate a related set of RP scores into a single distribution. Study 1 demonstrates how researcher degrees of freedom in mediation analysis, such as trying many different mediators and outcomes or testing moderated mediation, can inflate false-positive rates to up to 20% in the scenarios we examined. This study also demonstrates the validity of the RP index by examining its correspondence with false-positive findings. In Study 2, we compare the RP distribution of mediation analyses published in top marketing journals (2018-2020) to simulations of null effects, low power, and adequate power. We find that mediation analyses in marketing substantially deviated from simulations with adequate power but were similar to simulated distributions with low power and null effects. We propose study preregistration, increased statistical power, and reduction of selection bias as solutions to increase evidential value of mediation analyses going forward.