Predicting psychological constructs from biased measurements: The impact of non-invariant targets in machine learning

Psychology is increasingly interested in the prediction of constructs via machine learning (ML) models, for example, predicting a person’s personality or intelligence. To measure these constructs, psychologists often draw on questionnaires. In supervised ML, these measurements are then used as target variables (i.e., the “ground truth”) for model training. It is currently paid only little attention to psychometric issues and biases that might be carried over from measurements in the training data to the final model used for predictions. One potential bias is a lack of measurement invariance (MI) of the questionnaire data across groups that are used as target values for supervised learning. If non-invariant measrurements are used for model training, this might bias the predictions of the final ML model. Specifically, people from two different groups with the same true score on a construct might receive different predicted scores by the model. In this article, we assess the impact of a lack of MI in target variables on ML predictive performance and investigate approaches to counter this impact. We address this question by a comprehensive simulation study in which we derive target values from (a) single-group models (i.e., ignoring non-invariance) and (b) alignment optimization (i.e., handling non-invariance). Results show that single-group factor scores make ML models reproduce measurement bias in their predictions. Aligned factor scores can improve prediction performance if measurements are non-invariant, but only if certain conditions are met. We discuss implications for psychological applications of ML as well as directions for future research.