Unmeasured confounding and misclassification in vaccine effectiveness studies using electronic health records (EHRs): an evaluation of a multi country European study (VEBIS-EHR)

Background

Electronic health record (EHR)-based observational studies can rapidly provide real-world data on vaccine effectiveness (VE), particularly key during the COVID-19 pandemic. However, EHR data may be prone to misclassification and unmeasured confounding, requiring systematic mitigation to ensure robust findings.

Methods

In VEBIS-EHR, a retrospective multi-country COVID-19 VE cohort study, we examined unmeasured confounding using a negative control outcome (death not related to COVID-19) and misclassification from varying data extraction intervals. The evaluation spanned two periods (November-December 2023, January-February 2024), encompassing up to 18.7 million individuals across six EU/EEA countries. Vaccine confounding-adjusted hazard ratios (aHRs) were pooled using random-effects meta-analysis.

Results

aHRs against non–COVID-19 mortality ranged from 0.35 (95% CI: 0.28–0.44) to 0.70 (0.66–0.73) when comparing vaccinated versus unvaccinated. Delaying EHR data extraction modestly increased the capture of outcome and exposure events, with some variation by vaccination status. Site-level fluctuations in aHRs did not meaningfully alter the overall pooled VE, suggesting stable estimates despite misclassification related to extraction timing.

Conclusions

We observed some evidence of unmeasured confounding when using non-COVID-19 deaths as a negative outcome, though the specificity of our negative control must be considered. This result may suggest overestimation of VE, but also the need for further analysis with more specific negative control outcomes and confounding-adjustment techniques. Addressing such confounding using richer data sources and more refined approaches remains critical to ensure accurate, timely VE estimates when using real-world EHR-based data. Extending the delay between the end of observation and data extraction modestly improves the completeness of exposure and outcome data, with limited effect on pooled VE estimates.

Key Messages

A Key Messages section should be added after the keywords and before the article’s introduction, with the key messages of the paper made in 3 bullet points that succinctly describe:

• What your research question was: Whether vaccine effectiveness estimates based on EHR data are internally valid according to analyses focused on two topics of concern related to EHR-based observational research – unmeasured confounding and misclassification.

• What you found: We observed unmeasured confounding in our estimates when using non-COVID19 deaths as negative outcome, and timing of data abstraction from source EHRs was found to have a small influence on the capture and classification of exposure and outcome.

• Why it is important: Our results reinforce earlier evidence of the healthy vaccinee phenomenon indicating possible biases in the estimates of vaccine effectiveness estimates obtained from EHR data and that there appears to be little influence of EHR extraction timing on data completeness.