Text-level effects of artificial intelligence-mediated language refinement on author–reader alignment and comprehension in Chinese non-academic and academic texts

This study examined whether target-informed, author-guided large language model (LLM) refinement alters author-reader alignment and comprehension in Chinese texts. Two controlled experiments compared original texts with versions refined by GPT-4 accessed through ChatGPT, Claude 3 Opus, and, in Study 1, Gemini Advanced. Models received author-provided target ratings or answer-key information that was also used for outcome scoring, making the intervention criterion-linked rather than ordinary copyediting. Participants were recruited through Credamo and randomly allocated by the platform to text-version conditions. Study 1 (N = 572) measured exact-match alignment with authors’ intended emotional-expression ratings and accuracy on 15 author-keyed comprehension items. Recruitment was extended after inspection of initial results; its inferential analyses are therefore exploratory. The GPT-4 condition showed higher emotional-expression alignment than the original-text condition (p = .002, signed r = .18). Gemini Advanced reduced alignment at the unadjusted level (p = .010, signed r = − .15) but did not meet the Bonferroni-adjusted threshold of 0.0083; Claude 3 Opus did not differ from original texts. No comprehension differences were detected. Study 2 (N = 420) measured exact-match alignment on six textually conveyed scholarly attributes and author-keyed comprehension. Neither GPT-4 nor Claude 3 Opus produced detectable differences on either outcome. Results are conditional on the fixed materials, participant samples, prompts, single-output procedure, criterion-linked guidance, and scoring rules and do not establish preservation of all authorial meaning or objective scholarly quality.