Understanding the evolution of individual response process: An exploratory approach

The massive use of computer-adaptive testing has contributed to revolutionizing psychometric analysis in terms of both data collection and analysis methodologies. The National Assessment of Educational Progress (NAEP) is one of the most well-known assessments that collect respondents' process data, which has been extensively analyzed (Bosch, 2021; Levin, 2021; Patel et al., 2021; Zehner et al., 2020). Much of this work has primarily focused on efficiency estimation, due to the 2019 NAEP Educational Data Mining Competition. Prediction models were trained mainly using item-level feature engineering and person-level grouping. Some work focusing on exploring the data itself rather than training models has also focused on grouping respondents' behavior patterns on subsets of items (Wei et al., 2024) through the use of clustering and relating these patterns to demographics and respondent ability. With a similar goal, we expand on this approach by firstly developing \textit{item level} behavior profiles using clustering-based methods (i.e., latent class analysis) for all items. Using these profiles, we explore individual trajectories from profile to profile over the course of the test. Combining these trajectories with latent abilities and speed estimated from item responses and demographic information, we characterize the respondents who comprise the individual trajectories. Preliminary results find item-to-item variation in the structure and interpretation of the behavior profiles, but also distinct profiles within each item that represent differences in the underlying IRT-derived latent ability distributions. Understanding usage patterns for respondents of various ability levels, demographic backgrounds, and disability statuses may inform future item development, assistive tool design, and user interface for computerized assessments.