A general method for estimating reliability using Bayesian Measurement Uncertainty

Unreliable measurement can lead to lower statistical power, attenuated effect sizes and residual confounding. However, estimating reliability can be challenging for complex cognitive and behavioural assessments without test-retest data. Most statistical methods for assessing reliability with a single test administration are designed for fixed-item questionnaires. We introduce a novel Bayesian procedure called relative measurement uncertainty (RMU) for estimating reliability that can be broadly applied, including to many widely used computational models. Our approach draws pairs of samples from each subject's posterior and calculates the correlation between draws. We demonstrate analytically and via simulation that this method provides accurate reliability estimates and well-calibrated credible intervals across linear factor, signal detection and reinforcement learning models. Simulations found that RMU was more accurate (lower root mean square error) than existing measures (coefficient alpha, coefficient H, split-half), but more importantly, it can be applied across a wide range of computational models. RMU offers a general method for estimating reliability, leveraging the modelling flexibility of Bayesian statistical methods.