A comparison of the predictive performance of continuous and class-based latent trait models

The ability of a student can be conceptualized as either a continuously varying entity (e.g., conventional analysis using dichotomous item response theory (IRT) models; Lord and Novick, 1968) or a bundle of latent classes (e.g., cognitive diagnostic models (CDMs); Templin, Henson, et al., 2010; von Davier and Lee, 2019). This paper builds on recent efforts to focus on predictive differences between measurement models in an attempt to examine the degree to which such approaches—which utilize quite distinctive notions regarding the nature of ability—produce different predictions of response behavior in the real-world. We first present two simulation studies in which data are generated from variants of CDMs that differ in sample size, attribute hierarchical structures, and attribute estimation methods. We illustrate that, as we would expect given that they were used to generate data, CDM-based predictions uniformly outperform those of IRT models. We then compare the performance of CDM- and IRT-based approaches across nine empirical datasets previously analyzed using CDMs. Our findings indicate that overfitting is a pervasive issue across CDM-based predictions, particularly with the G-DINA model. Furthermore, only five out of nine datasets show improved model fit for CDMs over the 2PL model when using the marginal mastery probabilities estimator, and none show superior performance when using the maximum a posteriori estimator. Researchers and practitioners may need to balance the diagnostic appeal of CDMs with the fact that their complexity can come at the cost of predictive accuracy.