Error Classification in Stoichiometry Tutoring Systems with Different Levels of Scaffolding: Comparing Rule-Based Classification and Machine Learning

Comparing rule-based and machine learning (ML) approaches to error classification is crucial for advancing adaptive instruction. However, few studies have examined their comparative accuracy for tutoring systems with different levels of scaffolding. The present study addresses this gap by examining the classification of stoichiometry errors using data from 61 science students enrolled at a public German university who interacted with two distinct tutoring systems. We annotated 1,164 error clips from log data and derived an error classification scheme with eight categories covering system-related (e.g., usability) and domain-specific (e.g., unit conversion) categories. We developed decision rules and trained an ML model, comparing automatically classified errors in segments of learner inputs to classifications based on our expert model. Our results indicate that domain-specific errors requiring procedural knowledge are more accurately classified by the rule-based classifier, while concept-based errors are better captured by ML, though only in a lowly scaffolded tutoring system. These findings suggest researchers must carefully choose modeling approaches to address misconceptions in STEM learning.