Evaluating an LLM’s Performance in Annotating Discourse Strategies

Manual annotation remains essential for identifying complex pragmatic and discourse-level features in corpus linguistics, particularly the functional components of speech acts. While part-of-speech and semantic tagging can be automated with high accuracy, annotating discourse strategies remains challenging due to their context-sensitive nature and lack of consistent lexical realizations. These limitations hinder the scalability of function-to-form approaches and constrain the development of richly annotated corpora for pragmatics research and instruction. This study investigates whether a large language model (LLM), specifically ChatGPT-4, can support functional annotation of refusal strategies in English. A corpus of written Discourse Completion Tasks by Japanese university English learners was analyzed for reliability, human-rater agreement, accuracy, and generalizability. The results suggest an LLM can greatly assist the process of pragmatic annotation to increase scalability and accuracy.