Developing a Comprehensive Framework for Real-World Data Case Validation in Vaccine Safety Monitoring: The VAC4EU experience

Purpose Real-world evidence (RWE) is essential for post-licensure monitoring of vaccine safety, as pre-licensure trials are often limited in sample size, population diversity, and duration of follow-up. The Vaccine Monitoring Collaboration for Europe (VAC4EU) has developed a structured validation pipeline that operationalizes Brighton Collaboration (BC) case definitions for real-world data (RWD) in a harmonized, scalable and reusable manner. Methods VAC4EU developed a systematic, stepwise approach to validate vaccine safety outcomes utilizing BC case definitions where available. The approach involves: 1) Critical review of BC definition and adaptation to RWD by clinical and RWD experts; 2) Creation of dummy cases based on published reports; 3) Creation of REDCap electronic data collection forms (eDCF) incorporating decision logic to assigned levels of certainty (LOC); 4) Iterative testing of decision logic; and 5) comprehensive training of abstractors with real-time feedback. A dedicated task force assigned reference LOCs for dummy cases. Inter-rater reliability was measured using Fleiss' kappa (κ) by comparing abstractor LOCs to the reference standard. Results The validation pipeline was applied to 16 COVID-19 vaccine safety outcomes, 13 of them had existing BC definitions. Eleven eDCFs required adaptation to accommodate missing or unstructured information common in RWD. In total, 78 dummy cases were developed across the 16 outcomes, and 15 REDCap eDCFs were created. Myocarditis and pericarditis shared a single form. Across 33 trained abstractors, 747 individual case abstractions were completed. Agreement analysis showed 93 discrepancies (12.4%) and moderate overall concordance (κ = 0.55), with the lowest for Thrombosis with Thrombocytopenia Syndrome (κ = -0.05). Conclusion The VAC4EU validation pipeline provides a standardized framework for training and validating vaccine safety outcome validation using RWD. By adapting BC case definitions and emphasizing targeted abstractor training with ongoing feedback, this approach improves the reliability of post-marketing surveillance studies. Keywords: Validation, Vaccine safety, Real-world data, Brighton Collaboration, Post-authorization safety studies