Potentially harmful consequences of artificial intelligence (AI) chatbot use among patients with mental illness: Early data from a large psychiatric service system

Chatbots driven by generative artificial intelligence (AI chatbots) have become ubiquitous. Recently, concerns have risen over the possibility that use of AI chatbots may be harmful for people prone to mental illness. Here, we aimed to investigate whether there are reports compatible with potentially harmful consequences of AI chatbot use on mental health among patients with mental illness receiving care in a large psychiatric service system. Based on screening of clinical notes from electronic health records mentioning ‘chatbot’ or ‘ChatGPT’ from the Psychiatric Services of the Central Denmark Region, we report on 38 patients for whom use of AI chatbots had potentially harmful consequences for their mental health - with worsening/consolidation of delusions being the most common presentation. Mental health professionals should be aware of this possibility and guide their patients accordingly, as it seems that some patients would likely benefit from reduced/no use of AI chatbots in their current form.