The Effect of Pandemic Prevalence on the Reported Efficacy of SARS-CoV-2 Vaccine Candidates: A Systematic Review and Meta-analysis

  1. Rajeev Sharma
  2. Abhijith Anand

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

Importance

The efficacy of SARS-CoV-2 vaccine candidates reported in Phase 3 trials varies from ∼45% to ∼95%. It is important to explain the reasons for this heterogeneity.

Objective

To test the hypothesis that the efficacy of SARS-CoV-2 vaccine candidates falls with increasing prevalence of the COVID-19 pandemic.

Data Sources

ClinicalTrials.gov , WHO, McGill and LSHTM trackers of COVID-19 candidate vaccines, peer reviewed publications, and press releases were searched until March 31 st , 2021.

Study Selection

All RCTs reporting efficacy outcomes from Phase 3 trials till March 31 st , 2021 were included. Of the 11 vaccine candidates that had started their Phase 3 trials by November 1, 2020. Phase 3 efficacy outcomes were available for 8 vaccine candidates. (PROSPERO CRD42021243121).

Data Extraction and Synthesis

Both authors independently extracted the data required from identified sources, using PRISMA guidelines. The analysis included all RCTs reported in peer reviewed publications and publicly available sources. A random effects model with restricted maximum likelihood estimator was used to summarize the treatment effects. Cochrane Risk of Bias Assessment Tool was used to assess risk of bias. Certainty of evidence was assessed using the GRADE tool.

Main Outcomes and Measures

SARS-CoV-2 infections per protocol in vaccine and placebo groups, risk ratio, prevalence of the COVID-19 infection rate in the populations where the Phase 3 trials were conducted.

Results

8 vaccine candidates had reported efficacy data from a total of 20 independent Phase 3 trials, representing a total of 221,968 subjects, 453 infections across the vaccinated groups and 1,554 infections across the placebo groups. The overall estimate of the risk-ratio is 0.24 (95% CI, 0.17-0.34, p < 0.01), with an I 2 statistic of 88.73%. The meta-regression analysis with pandemic prevalence as the moderator explains almost half the variance in risk ratios across trials (R 2 =49.06%, p<0.01).

Conclusion and Relevance

Pandemic prevalence explains almost half of the between-trial variance in reported efficacies. Efficacy of SARS-CoV-2 vaccine candidates declines as the pandemic prevalence increases.

Key Points

Question

Does the prevalence of the COVID-19 pandemic explain the heterogeneity in efficacies reported across Phase 3 trials of SARS-CoV-2 vaccine candidates?

Findings

Almost 50% of the variance in efficacies reported across Phase 3 trials can be explained by differences in COVID-19 infection rate prevailing across trials. Efficacy of evaluated SARS-CoV-2 vaccine candidates falls significantly with increasing prevalence of the COVID-19 pandemic across trial sites.

Meaning

Efficacy of SARS-CoV-2 vaccine candidates needs to be interpreted in conjunction with the prevalence of the COVID-19 pandemic. Adjustment for location-level prevalence analysis would provide better insights into the efficacy results of Phase 3 trials.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.06.05.21258394: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Ethicsnot detected.
    Sex as a biological variablenot detected.
    RandomizationVaccine candidates were included in this study if they reported efficacy results from Phase 3 randomised controlled trials.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    In particular, the WHO maintains a tracker database that compiles detailed information on the SARS-CoV-2 vaccine candidate landscape, tracks vaccine candidates in development, and regularly updates progress on registered trials that are underway: “To ensure the latest information is available, the landscape will be updated twice a week (Tuesday and Friday, 17:00 CET) by searching, gathering and cross-checking data from multiple sources such as the Cochrane vaccine mapping tool, PubMed, ClinicalTrials.gov, WHO ICTRP and from a network of researchers and industry for new candidate vaccines by screening registered trials for clinical information.
    Cochrane vaccine mapping tool
    suggested: None
    PubMed
    suggested: (PubMed, RRID:SCR_004846)
    Risk of Bias Assessment: Both authors independently assessed the risk of bias using Cochrane Risk of Bias Assessment Tool.
    Cochrane Risk
    suggested: (Robot Reviewer, RRID:SCR_018961)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Limitations: The overall sample is small, 20 trials across 8 vaccine candidates. Efficacy data on three other vaccine candidates that have completed Phase 3 trials and are currently approved for use is not publicly available and could not be included in this meta-analysis. All trials are multi-location, and a few are multi-country too, e.g., Pfizer/BioNTech’s trial conducted in the US, Chile and Peru (see Table 2, Note a). In the absence of data on the exact locations of those trials and the number of subjects in the trial at each location, we have used country-level data to rank each trial on pandemic prevalence. This assumption necessarily introduces an element of error in estimating pandemic prevalence. To some extent, that effect is mitigated in our meta-regression by employing rank of pandemic prevalence as the moderator variable. Future Phase 3 trials could be required to report a location-level analysis to provide better insights into the efficacy results of Phase 3 trials.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.06.05.21258394 on medRxiv