The Ability of ChatGPT to Aid in the Rapid Development of Inoculation Message Treatments: A Case Study and Recommendations

One of the most widely known types of AI technologies in recent years is ChatGPT. ChatGPT is a state-of-the-art language model that has revolutionized natural language processing by generating human-like text with context and coherence, enabling new possibilities for human-AI interaction (Brown et al., 2020). This case study reports on a 10-week conversational AI training initiative to assess whether ChatGPT4 can (a) identify the structural elements and features of conventional inoculation messages (i.e., forewarnings and preemptive refutations); (b) augment inoculation message design features (i.e., linguistic signatures, language, and length); (c) adapt messages for specific audiences (i.e., younger populations, experts); and finally, (d) independently replicate the inoculation message development process on a topic of its choosing. Twenty-one different inoculation messages previously used in published inoculation studies provided the instructional foundation for teaching ChatGPT. A combination of prompting techniques were used (i.e., sequential, active, iterative, and chain of thought prompts) to achieve the goals of the study. Using 29 different prompts we found a high degree of originality within the AI generated inoculation messages; however, structural weaknesses were prevalent regardless of originality. AI generated messages were more difficult to read and required an advanced education level to comprehend. The AI messages were not equivalent with the training exemplars, and, in general, contained higher percentages of complex wording. We also found that ChatGPT struggled with developing the explicit forewarning threat component and did not generate inoculation messages with more than two refutations. Several themes of metaphorical and figurative language were used by the conversational-AI. We describe and contextualize these findings and discuss considerations and recommendations for future study.