Bot or not: Can people tell the difference between stories written by a human or by an AI system?

Given that artificial intelligence (AI) programs like ChatGPT can generate creative content, some fear that such programs will displace human writers. But can AI-generated creative works truly compete with human-authored works? In Study 1, participants (N=1,498 adults recruited from Prolific) read either a human-written or AI-generated story. Regardless of their story’s true origin, half of the participants were told that their story was written by a human, while the other half were told that their story was written by ChatGPT. After reading, participants rated the story’s quality and their level of absorption into the story. Participants rated the AI-generated story as higher in quality and more absorbing, but they rated the story better on both measures when they had been told it was written by a human. In Study 2, participants (N=424 adults recruited from Prolific) read both a human-written and AI-generated fictional short story and guessed each story’s origins. They also answered questions regarding their experiences with ChatGPT and other AI software and their experiences with fictional literature. While participants were significantly worse than chance at differentiating between the human-written and AI-generated stories, experience with AI software expertise were positively correlated with correct performance. Experience with fictional literature had no significant effect. The results from these studies suggest that AI programs might be able to generate human-level creative work but are not perceived as being able to do so.