Measuring Vaccine Efficacy Against Infection and Disease in Clinical Trials: Sources and Magnitude of Bias in Coronavirus Disease 2019 (COVID-19) Vaccine Efficacy Estimates
This article has been Reviewed by the following groups
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
- Evaluated articles (ScreenIT)
Abstract
Background
Phase III trials have estimated coronavirus disease 2019 (COVID-19) vaccine efficacy (VE) against symptomatic and asymptomatic infection. We explore the direction and magnitude of potential biases in these estimates and their implications for vaccine protection against infection and against disease in breakthrough infections.
Methods
We developed a mathematical model that accounts for natural and vaccine-induced immunity, changes in serostatus, and imperfect sensitivity and specificity of tests for infection and antibodies. We estimated expected biases in VE against symptomatic, asymptomatic, and any severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infections and against disease following infection for a range of vaccine characteristics and measurement approaches, and the likely overall biases for published trial results that included asymptomatic infections.
Results
VE against asymptomatic infection measured by polymerase chain reaction (PCR) or serology is expected to be low or negative for vaccines that prevent disease but not infection. VE against any infection is overestimated when asymptomatic infections are less likely to be detected than symptomatic infections and the vaccine protects against symptom development. A competing bias toward underestimation arises for estimates based on tests with imperfect specificity, especially when testing is performed frequently. Our model indicates considerable uncertainty in Oxford-AstraZeneca ChAdOx1 and Janssen Ad26.COV2.S VE against any infection, with slightly higher than published, bias-adjusted values of 59.0% (95% uncertainty interval [UI] 38.4–77.1) and 70.9% (95% UI 49.8–80.7), respectively.
Conclusions
Multiple biases are likely to influence COVID-19 VE estimates, potentially explaining the observed difference between ChAdOx1 and Ad26.COV2.S vaccines. These biases should be considered when interpreting both efficacy and effectiveness study results.
Article activity feed
-
-
SciScore for 10.1101/2021.07.30.21260912: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your code.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:There are limitations to our analysis, notably uncertainties over parameter estimates. There is limited evidence on both serology and PCR test sensitivities for asymptomatic infections, and how these change over time since infection. As we show, differences in test sensitivity by symptom status can lead to overestimation of VEin, so further studies are needed to clarify the potential role of this bias. We also did not consider the vaccines’ …
SciScore for 10.1101/2021.07.30.21260912: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your code.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:There are limitations to our analysis, notably uncertainties over parameter estimates. There is limited evidence on both serology and PCR test sensitivities for asymptomatic infections, and how these change over time since infection. As we show, differences in test sensitivity by symptom status can lead to overestimation of VEin, so further studies are needed to clarify the potential role of this bias. We also did not consider the vaccines’ effects on viral load and how this alters virological and serological test sensitivity. Multiple COVID-19 vaccines have been shown to reduce SARS-CoV-2 viral load (33,34), and lower load infections are less likely to lead to seroconversion (35). Therefore serology-based efficacy estimates may be more representative of high viral load infections than all infections. They may be comparable to estimates based on DNA sequenced swabs, as these samples must exceed a threshold viral load to be sequenced. Finally, we do not consider the use of point prevalence estimates from single time point PCR swabs, however this has been explored elsewhere (11,12). In conclusion, multiple biases have the potential to influence the COVID-19 vaccine efficacy estimates, with their direction and magnitude dependent on the vaccine properties and testing strategies. These biases may explain differences between the ChAdOx1 and Ad26.COV2.S trial reported estimates despite similar vaccine platform technologies, and should be considered when interpreting both efficacy a...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
Results from scite Reference Check: We found no unreliable references.
-
