From Ells to Metres: Population norms should supersede sample-local standardisation

In the clinical and behavioural sciences, researchers often report transformed effect size measures like correlation coefficients and Cohen’s ds, purportedly to facilitate interpretability and comparability to other studies or treatment trials. These values are often referred to as standardised and treated as unitless when they are relative to the sample-local estimate of variability. Variances differ from sample to sample for reasons unrelated to the size of the effect, such as selection into the sample and measurement error. Heterogeneity in the local sample variances will in turn introduce heterogeneity into the effect sizes using these variances as standardizers when no such heterogeneity exists in the raw effects. We argue that replacing local with common standards for the purpose of standardisation would improve the interpretability and comparability of results. Whenever available, test norms (i.e., estimates of the mean and standard deviation of the outcome measure for a given target population) could serve as the field-wide common standards to which smaller study samples are compared and calibrated. Concretely, reports of sample descriptive statistics would include the average norm-standardised scores, and effects would be reported in those norm-standardised units. Changing reporting in this manner would make it easy to diagnose selection bias at a glance in addition to yielding more interpretable and comparable effect sizes. In the long term, we need to invest more resources into deriving test norms to increase the availability of high-quality common standards. In the short term, we argue that even imperfect common standards are better than local ones.