COVID-19 vaccination effectiveness rates by week and sources of bias: a retrospective cohort study

  1. Anna Ostropolets
  2. George Hripcsak

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

To examine COVID-19 vaccine effectiveness over six 7-day intervals after the first dose and assess underlying bias in observational data.

Design and setting

Retrospective cohort study using Columbia University Irving Medical Center data linked to state and city immunisation registries.

Outcomes and measures

We used large-scale propensity score matching with up to 54 987 covariates, fitted Cox proportional hazards models and constructed Kaplan-Meier plots for two main outcomes (COVID-19 infection and COVID-19-associated hospitalisation). We conducted manual chart review of cases in week 1 in both groups along with a set of secondary analyses for other index date, outcome and population choices.

Results

The study included 179 666 patients. We observed increasing effectiveness after the first dose of mRNA vaccines with week 6 effectiveness approximating 84% (95% CI 72% to 91%) for COVID-19 infection and 86% (95% CI 69% to 95%) for COVID-19-associated hospitalisation. When analysing unexpectedly high effectiveness in week 1, chart review revealed that vaccinated patients are less likely to seek care after vaccination and are more likely to be diagnosed with COVID-19 during the encounters for other conditions. Secondary analyses highlighted potential outcome misclassification for International Classification of Diseases, Tenth Revision, Clinical Modification diagnosis, the influence of excluding patients with prior COVID-19 infection and anchoring in the unexposed group. Long-term vaccine effectiveness in fully vaccinated patients matched the results of the randomised trials.

Conclusions

For vaccine effectiveness studies, observational data need to be scrutinised to ensure compared groups exhibit similar health-seeking behaviour and are equally likely to be captured in the data. While we found that studies may be capable of accurately estimating long-term effectiveness despite bias in early weeks, the early week results should be reported in every study so that we may gain a better understanding of the biases. Given the difference in temporal trends of vaccine exposure and patients’ baseline characteristics, indirect comparison of vaccines may produce biased results.

Article activity feed

  1. Version published to 10.1136/bmjopen-2022-061126
  2. ScreenIT

    SciScore for 10.1101/2021.12.22.21268253: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    All analyses were supported by the OHDSI Infrastructure (CohortMethod package, available at https://ohdsi.github.io/CohortMethod/, FeatureExtraction available at https://ohdsi.github.io/FeatureExtraction/ and the Cyclops package for large-scale regularized regression 19 available at https://ohdsi.github.io/Cyclops).
    CohortMethod
    suggested: (CohortMethod, RRID:SCR_018511)

    Results from OddPub: Thank you for sharing your code.

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.12.22.21268253v1 on medRxiv