Exploring the Impact of Non-Linearity on Effect Heterogeneity in Experimental Data.

Heterogeneity has been proposed as a framework to improve psychological and behavioral science by identifying mechanisms that explain the differences between replications of an effect. The modal response to such differences is to search for relevant moderators—typically with limited success. We argue that monotonic non-linear relationships can generate heterogeneous effects across replications when baseline levels differ, even in the absence of moderators. To evaluate this possibility, we introduce four descriptors of heterogeneity: effect heterogeneity, control group heterogeneity, the intercept–slope correlation, and the rank correlation between control and treatment means. These characterize patterns consistent with linear, concave, or convex functional relationships. We apply this analytical framework to an initial pool of 55 meta-analytic multi-lab data sets. All data sets with heterogeneous effects also demonstrated control group heterogeneity, and several patterns were in line with a concave functional relationship. In some cases, the association between baselines and effects accounted for a substantial proportion of effect heterogeneity. Additionally, only a small number of data sets were directly at odds with a monotonic non-linear functional form. Moreover, our results suggest that effect heterogeneity in multi-lab research reflects diverse data patterns rather than providing support for treating moderators as the modal explanation for effect heterogeneity. We discuss implications for multi-lab and meta-analytic practice, study design, and interpretations of effect heterogeneity. We advocate for a shift from treating only the effect size distribution as the explanandum of the heterogeneity framework to more elaborate descriptions of heterogeneity.