Comprehending semantic and syntactic anomalies in LLM- versus human-generated texts: An ERP study

As large language models (LLMs) become increasingly proficient at engaging in human-like conversations, it is essential to understand how people process language generated by LLMs compared to language produced by humans. During language comprehension, people interpret incoming linguistic input by integrating it with their world knowledge (e.g., semantic anomalies can elicit an N400 effect in brain potentials) and linguistic knowledge (e.g., syntactic anomalies can lead to a P600 effect). Crucially, people adjust their language comprehension based on the perceived demographic attributes of the speaker, which has been shown to modulate both semantics-sensitive N400 and syntax-sensitive P600 effects. In two ERP experiments, we investigated whether people are sensitive to the fact that LLMs excel in linguistic formulation (i.e., consistently producing grammatically correct texts) but are prone to hallucination (i.e., occasionally generating nonsensical content). Participants were informed that they would be reading texts previously generated by either an LLM or a human. Experiment 1 revealed an N400 effect for semantically anomalous sentences compared to semantically coherent ones. Importantly, the N400 effect was smaller for LLM-generated texts than for human-generated texts. Furthermore, participants who more strongly believed that LLMs possess human-like knowledge exhibited a larger N400 effect. Experiment 2 demonstrated a P600 effect for syntactic anomalies in LLM-generated texts, with the effect being larger for LLM-generated texts than for human-generated texts (and not statistically significant in the latter). These findings suggest that people’s expectations about LLMs’ potential for hallucination and near-perfect grammatical competence modulate the way they comprehend LLM-generated texts. This research highlights the importance of considering perceived language model attributes when studying human language comprehension in the context of AI-generated texts.