Enhancing Relation Extraction for COVID-19 Vaccine Shot-Adverse Event Associations with Large Language Models

Objective The rapid evolution of the COVID-19 virus has led to the development of different vaccine shots, each designed to combat specific variants and enhance overall efficacy. While vaccines have been crucial in controlling the spread of the virus, they can also cause adverse events (AEs). Understanding these relationships is vital for vaccine safety monitoring and surveillance. Methods In our study, we collected data from the Vaccine Adverse Event Reporting System (VAERS) and social media platforms (Twitter and Reddit) to extract relationships between COVID-19 vaccine shots and adverse events. The dataset comprised 771 relation pairs, enabling a comprehensive analysis of adverse event patterns. We employed state-of-the-art GPT models, including GPT-3.5 and GPT-4, alongside traditional models such as Recurrent Neural Networks (RNNs) and BioBERT, to extract these relationships. Additionally, we used two sets of post-processing rules to further refine the extracted relations. Evaluation metrics including precision, recall, and F1-score were used to assess the performance of our models in extracting these relationships accurately. Results The most commonly reported AEs following the primary series of COVID-19 vaccines include arm soreness, fatigue, and headache, while the spectrum of AEs following boosters is more diverse. In relation extraction, fine-tuned GPT-3.5 with Sentence-based Relation Identification achieved the highest precision of 0.94 and a perfect recall of 1, resulting in an impressive F1 score of 0.97. Conclusion This study advances biomedical informatics by showing how large language models and deep learning models can extract relationships between vaccine shots and adverse events from VAERS and social media. These findings improve vaccine safety monitoring and clinical practice by enhancing our understanding of post-vaccination symptoms. The study sets a precedent for future research in natural language processing and biomedical informatics, with potential applications in pharmacovigilance and clinical decision-making.