DEDICAITE - DEtecting AI-generated TExts in a DIdactic Context

The widespread use of generative AI by students poses challenges for university teachers. Recent studies showed that medical and humanities scholars familiar with student-written texts are 70% able to recognize whether a text was student-written or generated by ChatGPT. In a randomized study, we examine whether we can reproduce this hit-rate with a larger sample of teachers from all university faculties, and whether we can confirm the hypothesis that linguistic features rather than content are crucial for correct classification. Therefore, 295 university teachers received one of two samples of an academic text speaking e.g., about legal or a scientific topic, written either by a student or generated by ChatGPT-4.0. The participants were randomly assigned to two groups: one received detailed instructions on linguistic features for authorship recognition, the other did not. We then asked participants how familiar they were with the topic of the text (6-point-Likert) and whether the text was characterized by detailed argumentation, avoidance of redundancies, and a recurring theme. The detection rate was 66% and 63.8% (not significant), in both groups respectively, although only 11% had received a text with a familiar topic. For non-humanities scholars, the dedicated instructions led to a significantly higher hit rate (75% vs. 59%). In general, texts written by humans were more often correctly identified (72% vs. 58%). For the subsequent questions on text properties, the uni-variate analysis of the answers for correctly recognizing student texts, resulted in a highly significant positive agreement, and a rejection for ChatGPT-generated texts.