AI as a signal assessor – Can a Large Language Model perform causality assessment on a case series?

Background

Large Language Models (LLMs) are increasingly explored for pharmacovigilance tasks, including information extraction, case documentation, and single-case causality assessment. However, their ability to support causality assessment at the case series level — a complex, time-intensive task requiring clinical reasoning across multiple reports — remains unexplored.

Objective

To investigate how a large-scale general-purpose LLM can support pharmacovigilance professionals in assessing causality in a case series, and to explore how prompt design influences the quality of the model’s reasoning.

Methods

GPT-4o was used to assess causality for five drug – adverse event combinations, using an adaptation of the Bradford Hill viewpoints for case series assessment. The combinations represented varying drugs and vaccines, adverse events, and case series sizes (5–402 reports). One combination served as a negative control. Structured prompts were iteratively developed and refined using one combination, then applied to all combinations. LLM-generated assessments for each viewpoint were qualitatively evaluated by human annotators for accuracy (precision), and the LLM’s coverage of key aspects from the original signal text was assessed for one combination (recall).

Results

Across all five combinations, annotators agreed with 79–92% of the LLM’s output sentences. Full disagreement was consistently low (3–7%), with errors typically involving misinterpretation of complex report details rather than outright fabrication. Prompt design substantially influenced output quality; providing Bradford Hill viewpoint descriptions, including case series data, and adding explicit anti-hallucination instructions improved specificity and grounding. For the recall assessment, 15 of 23 key segments from the original signal text were reflected in the LLM output. The overall summary assessments demonstrated balanced reasoning, correctly distinguishing between positive safety signals and the negative control, and provided a coherent synthesis suitable as a starting point for human assessors.

Conclusions

LLMs have the potential to generate contextually nuanced and largely accurate preliminary causality assessments of case series aligned with the Bradford Hill viewpoints, with a low but non-zero hallucination rate. These findings support LLMs as a tool to augment, not replace, expert judgment in signal assessment. Future work should address larger and more diverse signal sets, improved evaluation frameworks for generative output, and the integration of pre-computed summary statistics to reduce errors.

Key points

• A large language model (GPT-4o) can support pharmacovigilance professionals by generating preliminary causality assessments of a case series that align with human expert judgment.

• Generated in minutes, such assessments could give assessors a structured starting point for in-depth review, condensing case-level information and surfacing early evidence for or against a signal.

• Output quality depended heavily on prompt design, and because the hallucination rate was low but not zero, these tools should augment, not replace, expert judgment in signal assessment.