Experiment-based calibration in psychology: foundational and data-generating model

Experiment-based calibration is a novel method for measurement validation, which – unlike classical validity metrics – does not require stable between-person variance. In this approach, the latent variable to be measuredis manipulated by an experiment, and its predicted scores – termed standard scores – are compared against the measured scores. Previous work has shown that under plausible boundary conditions, the correlationbetween standard and measured scores – termed retrodictive validity – is informative about measurement accuracy, i.e. combined trueness and precision. Here, I expand these findings in several directions. First, Iformalise the approach in a probability-theoretic framework with the concept of a standardised calibration space. Second, I relate this framework to classical validity theory and show that the boundary conditionsin fact apply to any form of criterion validity, including classical convergent validity. Thus, I state precise and empirically quantifiable boundary conditions under which criterion validity metrics are informative onvalidity. Third, I relate these boundary conditions to confounding variables, i.e. correlated latent variables. I show that in the limit, calibration will converge on the latent variable that is most closely related to thestandard. Finally, I provide a framework for modelling the data-generating process with Markov kernels, and identify sufficient conditions under which the data generation model results in a calibration space. In sum,this article provides a formal probability-theoretic framework for experiment-based calibration and facilitates modelling and empirical assessment of the data generating processes.