When and How Does LLM-Generated Feedback Surpass Traditional Automated Writing Evaluation? A Learning Trajectory Analysis of Writing Improvement

Although large language models (LLMs) have attracted growing attention as tools for educational feedback, their advantages over existing automated systems remain uncertain. This study compared the efficacy of LLM-generated formative feedback against a well-established automated writing evaluation (AWE) system in supporting English as a Foreign Language (EFL) writing. Using a randomized controlled design, the study recruited 60 EFL students who received feedback from either ChatGPT or Pigai across IELTS writing tasks over four weeks. Participants were blind to the feedback source, which was mediated by a research assistant. ChatGPT feedback employed evidence-informed prompt engineering addressing lower- and higher-order concerns with motivational language, whereas Pigai provided standard automated corrective feedback. Writing performance trajectories, analyzed using linear mixed-effects models on criterion-based IELTS scores, revealed a significant Time × Group interaction. While early-stage improvements were comparable, the ChatGPT group demonstrated sustained gains in the later phase, whereas the Pigai group plateaued. Additionally, the ChatGPT group showed significantly greater increases in writing motivation, particularly regarding self-efficacy and beliefs. Mediation analyses indicated that this motivational change partially mediated the association between feedback condition and late-phase writing improvement. No group differences were found in perceived feedback quality. These findings suggest that the advantages of LLM-based feedback over traditional AWE systems emerge temporally and are driven partially by motivational mechanisms. When implemented with pedagogically grounded prompt design, LLMs offer an effective and scalable approach to formative feedback in learning.