Unstable Measures, Unreliable Effects: Re-evaluating Replicability with Reliability Informed Confidence Intervals

Evaluating the replicability of findings in the behavioral sciences requires addressing the often-overlooked issue of measurement stability. Recent research highlights that instability is the norm for most psychological traits and behaviors. This study investigates the role of reliability of assessments and an avenue to address measurement instability with implications for replication efforts. Using measurement theory and simulated data, an adjustment to effect size confidence intervals is proposed by incorporating varying reliability estimates. Through simulations, an empirical test on data from two independent samples, and analyses on 97 effect sizes replicated in the Reproducibility Project: Psychology, Reliability-Informed Bayesian Credible Confidence Intervals (RIB CCI’s) are shown to yield more valid effect size confidence intervals. Findings highlight the critical role of reliability (i.e., measurement stability) in attenuation or inflation of effect sizes, effectively distorting expectations for replication. By integrating stability into effect size estimation, the approach offers a pathway towards more accurate research conclusions in human sciences, as well as any domain where there is inherent variability of the measurement.