Effects of non-pharmaceutical interventions on COVID-19: A Tale of Three Models

  1. Vincent Chin
  2. John P.A. Ioannidis
  3. Martin A. Tanner
  4. Sally Cripps

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

In this paper, we compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from three SIR models, all developed by the Imperial College COVID-19 Response Team. One model was applied to European countries and published in Nature 1 (model 1), concluding that complete lockdown was by far the most effective measure, responsible for 80% of the reduction in R t , and 3 million deaths were avoided in the examined countries. The Imperial College team applied a different model to the USA states 2 (model 2), and in response to our original submission, the Imperial team has proposed in a referee report a third model which is a hybrid of the first two models (model 3). We demonstrate that inference is highly nonrobust to model specification. In particular, inference regarding the relative effectiveness of NPIs changes substantially with the model and decision makers who are unaware of, or ignore, model uncertainty are underestimating the risk attached to any decisions based on that model. Our primary observation is that by applying to European countries the model that the Imperial College team used for the USA states (model 2), complete lockdown has no or little effect, since it was introduced typically at a point when R t was already very low. Moreover, using several state-of-the-art metrics for Bayesian model comparison, we demonstrate that model 2 (when applied to the European data) is better supported by the data than the model published in Nature 1 . In particular, serious doubt is cast on the conclusions in Flaxman et al. 1 , whether we examine the data up to May 5th (as in Flaxman et al. 1 ) or beyond the point when NPIs began to be lifted. Only by objectively considering a wide variety of models in a statistically principled manner, can one begin to address the effectiveness of NPIs such as lockdown. The approach outlined in this paper provides one such path.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.07.22.20160341: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: Thank you for sharing your code.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Some limitations of our work should be acknowledged. Besides model fit and parsimony metrics, theoretical and subjective considerations, as well as experience from other countries should be considered in model choice. However, given the observational nature of the data and the dynamic course of epidemic waves, one should avoid strong priors about effectiveness of different NPIs. Similarly, our results should not be interpreted with a nihilistic lens, i.e. that NPIs are totally ineffective. Decreasing exposures makes sense as a way to reduce epidemic wave propagation and eventually fatalities. However, if exposures can be reduced with less aggressive measures and fewer or no harms, this would be optimal. Finally, we did not examine very long-term time horizons. In theory, even effective measures may achieve only temporary mitigation and epidemic waves may surge again, when measures are relieved. We did observe this for the uplifting of measures in the July 12th analyses and empirical data from the emergence of second waves in many European countries and the USA in the fall of 2020 validate this hypothesis31. Availability of effective and safe vaccines may also affect risk-benefit ratios of NPI measures of different aggressiveness and different duration of implementation. Overall, observational data that feed into complex epidemic models should be dissected very carefully and substantial uncertainty may remain despite the best efforts of modelers28;32. While there has been resi...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2020.07.22.20160341v3 on medRxiv
  3. Version published to 10.1101/2020.07.22.20160341v2 on medRxiv
  4. Version published to 10.1101/2020.07.22.20160341v1 on medRxiv