A Machine Learning Pipeline to Classify the Type and Severity of Misspecifications in Latent Measurement Models

Evaluating and, if necessary, revising factor-analytic latent measurement models is a challenging task. By now, researchers can select from a toolbox of traditional and novel methods to evaluate global and local model fit. However, this toolbox lacks a method to evaluate model fit at an intermediate level facilitating a thoughtful revision process. In this study, we fill this gap with a machine learning (ML) approach to model fit evaluation. Specifically, we introduce an ML pipeline to classify the type and severity of misspecifications in multi-factorial measurement models. To build this pipeline, we trained three ML models that predict a) if the measurement model is correctly specified or if a latent factor, cross-loadings, factor correlations, or residual correlations have been neglected, b) if any, how many cross-loadings have been neglected, and c) if any, how many residual correlations have been neglected. All three ML models performed well overall, with performance drops mainly resulting from misclassifications by ± 1 cross-loading and ± 1–2 residual correlations, respectively. Consequently, the feedback of the ML pipeline must not be implemented in an exact manner. Instead, it points the user to the most likely source of misfit and indicates if the measurement model is not, mildly, moderately, or severely misspecified. This information provides helpful guidance in deciding if and how to revise the measurement model, the measurement instrument, or its underlying theory. We provide an open-access R implementation of the ML pipeline, discuss how to use it within a multi-method approach, and make recommendations on how to use its feedback for the next revision steps.