Approaches for estimating COVID-19 vaccine effectiveness using observational data in administrative databases: a systematic review
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
COVID-19 vaccine policy relied on observational vaccine-effectiveness (VE) studies conducted amid rapid variant turnover, evolving schedules, and shifting surveillance, yielding substantial heterogeneity in methodological approaches across studies. Prior reviews emphasised pooled or variant-specific VE, with limited attention to how methodological practice varied across countries and over time. Yet, understanding the landscape of methodological practices used during this period is essential for identifying opportunities to improve VE study design and conduct in future pandemic responses. This review systematically characterises the methodological practices in registry-based observational COVID-19 VE studies (2021–2024), documenting patterns in study design, statistical approaches, and analytical choices to establish an empirical foundation for methodological development in pandemic vaccine evaluation.
Methods
We ran a PRISMA-guided search of PubMed and Embase (via Ovid) from inception to Oct 14, 2024, for peer-reviewed observational studies estimating COVID-19 VE in routine (non-trial) settings that leveraged administrative/registry data (e.g., immunisation registries, laboratory/PCR databases, EHR/claims, hospitalisation/mortality registries, national-ID–linked datasets) and reported sufficient methodological detail to classify design, estimator, treatment of time, adjustment/matching/weighting, and sensitivity/validation checks. We excluded randomised trials; studies without administrative/registry data or confined to specialised populations; non-English publications; and duplicate analyses of the same cohort/time window. Descriptive summaries are presented overall, by calendar year, and by World Bank income group.
Results
253 studies from 61 countries met eligibility; most were from high-income settings (187/253, 73.9%). The median publication lag was 257 days (IQR 157–421), lengthening from 141 days in 2021 to 673 in 2024, while median cohort size declined over time. Cohorts (46.6%) and test-negative designs (43.1%) dominated; target-trial emulations (2.0%) and quasi-experimental studies (1.2%) were uncommon. Logistic regression (56.1%) and Cox models (24.8%) comprised the majority of primary estimator. Adjustment emphasised demographics, comorbidity, calendar time, and geography; variables proximate to testing behaviour and exposure opportunity were less frequent. Most studies reported no matching/weighting (155/253, 61.2%); among those that did, exact matching predominated and weighting was rare. Sensitivity analyses were not described in 98/253 (38.7%) of studies. Endpoints concentrated on infection, hospitalisation, and mortality, while variant-resolved analyses waned as PCR testing and sequencing contracted.
Conclusions
Observational COVID-19 vaccine VE studies scaled rapidly where registries existed, but remained concentrated in high-income settings, relied on a narrow estimator set, and infrequently applied validity checks. Strengthening privacy-preserving linkages (including sequencing), aligning designs to target-trial principles with marginal weighting, and normalising a lean validity toolkit could enhance interpretability and policy relevance.
Funding
This research is supported by the National Research Foundation Singapore under its Clinician Scientist-Individual Research Grant (MOH-001572) and administered by the Singapore Ministry of Health’s National Medical Research Council. J.T.L. is supported by the Ministry of Education (MOE), Singapore Start-up Grant. L.E.W. is supported by the National Medical Research Council through the Clinician Scientist New Investigator Award.
