Diagnostic Assessment of Deep Understanding Using Cognitive Diagnostic Models: A Large-Scale Assessment to Promote the Use of Effective Learning Strategies

Recent educational goals have focused on achieving deep understanding and promoting the use of effective learning strategies. Previous studies showed that cognitive diagnostic models (CDMs) help diagnose students' depth of understanding. However, the approaches in specifying attributes (i.e., Q-matrix) to diagnose the depth of understanding remain unexplored; therefore, utilizing large-scale assessments to capture the general trends in the students' mastery of the depth of understanding is an uncharted area in this field. This study explores which attribute expression (i.e., linear hierarchy or polytomous attribute) is more appropriate, based on the CDM analysis of a large-scale mathematics assessment. Traditional CDM applications frequently rely on the Markov chain Monte Carlo (MCMC) method, which imposes a substantial computational burden when a dataset is large and a model used in data analysis is complex. As a computationally efficient alternative to the MCMC method, this study adopts the variational Bayesian (VB) method to mitigate computational challenges brought by the MCMC method and illustrate how the VB method can be used for the CDM analysis of a large-scale assessment. The results from real data analysis show that a Q-matrix employing a linear hierarchy is more appropriate than employing polytomous attributes. The estimation results based on linear hierarchical attributes suggest that less than 30% of the sampled students achieved a deep understanding of procedures/formulas, whereas more than half of the students achieved only a shallow understanding. Regarding the understanding of terms, less than 35% of the students achieved either shallow or deep understanding. These results may help design and improve future learning strategy instructions.