Effects of Confounding Bias in Coronavirus Disease 2019 (COVID-19) and Influenza Vaccine Effectiveness Test-Negative Designs Due to Correlated Influenza and COVID-19 Vaccination Behaviors

  1. Margaret K Doll
  2. Stacy M Pettigrew
  3. Julia Ma
  4. Aman Verma

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Background

The test-negative design is commonly used to estimate influenza and coronavirus disease 2019 (COVID-19) vaccine effectiveness (VE). In these studies, correlated COVID-19 and influenza vaccine behaviors may introduce a confounding bias where controls are included with the other vaccine-preventable acute respiratory illness (ARI). We quantified the impact of this bias on VE estimates in studies where this bias is not addressed.

Methods

We simulated study populations under varying vaccination probabilities, COVID-19 VE, influenza VE, and proportions of controls included with the other vaccine-preventable ARI. Mean bias was calculated as the difference between estimated and true VE. Absolute mean bias in VE estimates was classified as low (<10%), moderate (10% to <20%), and high (≥20%).

Results

Where vaccination probabilities are positively correlated, COVID-19 and influenza VE test-negative studies with influenza and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) ARI controls, respectively, underestimate VE. For COVID-19 VE studies, mean bias was low for all scenarios where influenza represented ≤25% of controls. For influenza VE studies, mean bias was low for all scenarios where SARS-CoV-2 represented ≤10% of controls. Although bias was driven by the conditional probability of vaccination, low VE of the vaccine of interest and high VE of the confounding vaccine increase its magnitude.

Conclusions

Where a low percentage of controls is included with the other vaccine-preventable ARI, bias in COVID-19 and influenza VE estimates is low. However, influenza VE estimates are likely more susceptible to bias. Researchers should consider potential bias and its implications in their respective study settings to make informed methodological decisions in test-negative VE studies.

Article activity feed

  1. Version published to 10.1093/cid/ciac234
  2. ScreenIT

    SciScore for 10.1101/2021.10.22.21265390: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    All analyses were conducted using RStudio with R version 4.1.0 (The R Foundation for Statistical Computing, Vienna, Austria).
    RStudio
    suggested: (RStudio, RRID:SCR_000432)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    While our findings may be viewed as reassuring regarding bias in COVID-19 test-negative VE studies, these results are subject to several important limitations. First, we caution that our simulations examine bias scenarios we considered likely in the general population based on empiric U.S. data; however, we did not explore plausible scenarios by subpopulation. It is possible that some subpopulations, such as older persons or persons who are at higher risk of severe disease, may have a higher conditional probability of COVID-19 vaccination, given influenza vaccination status; this difference could thus, cause greater bias than we observed in our estimates. Similarly, regional variation in COVID-19 and influenza vaccination coverage may also affect the conditional probability of vaccination. Currently, lower uptake of COVID-19 vaccination in some southern U.S. states more closely aligns with influenza vaccination coverage in these regions.16, 17 These differences may also increase the conditional probability of vaccination, and thus, represent a setting where greater bias in COVID-19 VE estimates can occur. Both examples demonstrate that bias in VE estimates may be differential by subpopulation, which may be important for the interpretation of VE results. Additionally, even where the conditional probability of vaccination is the same, we found that bias can vary by true VE. In the case of COVID-19, where true VE is likely to vary by vaccine product,4, 20 bias in VE estimates wi...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.10.22.21265390v1 on medRxiv