A Taylorizable Process for Textual Detector Development

Textual detectors of cognitive constructs play a central role in educational data mining, learning analytics, and automated assessment, enabling scalable analysis of student thinking, discourse, and learning processes. Recent work has increasingly adopted LLM-as-a-judge approaches, which replace traditional supervised learning with rubric-based prompting of large language models. While these approaches substantially reduce development effort, they are often deployed without the forms of rigor historically emphasized in EDM and related communities. At the same time, fully rigorous detector development remains costly and slow, creating pressure to adopt lower-quality shortcuts. In this paper, we propose a Taylorizable process for developing textual detectors of cognitive constructs using LLMs—attempting to reconcile methodological rigor with efficiency and scalability. Drawing on prior detector development practices and principles of scientific management, we decompose detector construction into a standardized, end-to-end workflow encompassing construct definition, human coding and reliability checking, LLM-based detector construction, evaluation, and application at scale. We argue that this process supports four essential forms of rigor—assessment of good-ness, generalizability, construct validity, and auditability—while enabling streamlined execution by mixed-expertise teams. By making rigorous detector development more reproducible, auditable, and efficient, this framework aims to raise methodolog-ical standards for LLM-based measurement while preserving the practical advantages, including feasibility and lower development effort, that have driven their rapid adoption.