Liability or reliability? Exploring the role of effect size attenuation on meta-analytic heterogeneity

It is a widely accepted fact that standardized effect sizes, such as Cohen's d, are affected by imperfect score reliability. Accordingly, it seems sensible to assume that differences in score reliability across replications of the same effect affect the heterogeneity of its standardized effect sizes. This relationship has been discussed exclusively as an inflating one, implying that differences in score reliability necessarily lead to larger effect size heterogeneity. In a re-analysis of data from twelve multi-site replication projects, we demonstrate that this is not the case. After correcting for differences in score reliability, we observe no reduction in effect size heterogeneity, instead, for some cases we observe a slight increase. We turn towards an alternative discussion of how score reliability may affect standardized effect sizes by describing these as random ratio variables. Additionally, we propose two metrics and an inequality that can help understand under which circumstances heterogeneity in standardized effect sizes is more likely to be reduced or increased, if differences in score reliability are corrected for.