A case study on norming an intelligence test using Regularised Prediction and Poststratification from a Total Survey Error perspective

Valid generalisations from sample to population require representative samples. This is especially the case for psychological test norms as they are only useful to the extent that the normative samples can stand in for reference populations. However, as a field, we continue to face difficulties defining, acquiring, and describing representative samples. Having engaged with similar problems for a long time, political and other social scientists have developed a sophisticated set of tools for dealing with them. In this article, we therefore draw on insights from these fields to illustrate what measures need to be taken at all research stages to maximise sample representativeness, and the potential benefits of sample weighting in psychology. We argue that a specific class of weighting methods, Regularised Prediction and Poststratification (RPP) holds promise for improving psychological test norming practice. RPP involves fitting a regularised prediction model to a large and diverse sample and weighting predicted scores by true population values (sourced from census data, for example). Using IQ test data from the TwinLife study (N = 9,980, Culture Fair Test, CFT 20-R) as a case study, we show that RPP yields IQ scores that differ by up to 19 IQ points (on average by 4.21 points) from those reported in the CFT 20-R manual, which were based on traditional norming methods. We observed sizeable differences to other norming/weighting approaches, such as semiparametric and other continuous norming methods. Differences of this magnitude can lead to decision errors with potentially life-altering consequences (e.g., about intellectual disability). We contend that psychology could benefit from more widely available, high-quality norms for many aspects of research and practice and plead for the change in research practice necessary to make these benefits possible.