The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories

  1. CMMID COVID-19 Working Group
  2. Yang Liu
  3. Christian Morgenstern
  4. James Kelly
  5. Rachel Lowe
  6. Mark Jit

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Background

Non-pharmaceutical interventions (NPIs) are used to reduce transmission of SARS coronavirus 2 (SARS-CoV-2) that causes coronavirus disease 2019 (COVID-19). However, empirical evidence of the effectiveness of specific NPIs has been inconsistent. We assessed the effectiveness of NPIs around internal containment and closure, international travel restrictions, economic measures, and health system actions on SARS-CoV-2 transmission in 130 countries and territories.

Methods

We used panel (longitudinal) regression to estimate the effectiveness of 13 categories of NPIs in reducing SARS-CoV-2 transmission using data from January to June 2020. First, we examined the temporal association between NPIs using hierarchical cluster analyses. We then regressed the time-varying reproduction number ( R t ) of COVID-19 against different NPIs. We examined different model specifications to account for the temporal lag between NPIs and changes in R t , levels of NPI intensity, time-varying changes in NPI effect, and variable selection criteria. Results were interpreted taking into account both the range of model specifications and temporal clustering of NPIs.

Results

There was strong evidence for an association between two NPIs (school closure, internal movement restrictions) and reduced R t . Another three NPIs (workplace closure, income support, and debt/contract relief) had strong evidence of effectiveness when ignoring their level of intensity, while two NPIs (public events cancellation, restriction on gatherings) had strong evidence of their effectiveness only when evaluating their implementation at maximum capacity (e.g. restrictions on 1000+ people gathering were not effective, restrictions on < 10 people gathering were). Evidence about the effectiveness of the remaining NPIs (stay-at-home requirements, public information campaigns, public transport closure, international travel controls, testing, contact tracing) was inconsistent and inconclusive. We found temporal clustering between many of the NPIs. Effect sizes varied depending on whether or not we included data after peak NPI intensity.

Conclusion

Understanding the impact that specific NPIs have had on SARS-CoV-2 transmission is complicated by temporal clustering, time-dependent variation in effects, and differences in NPI intensity. However, the effectiveness of school closure and internal movement restrictions appears robust across different model specifications, with some evidence that other NPIs may also be effective under particular conditions. This provides empirical evidence for the potential effectiveness of many, although not all, actions policy-makers are taking to respond to the COVID-19 pandemic.

Article activity feed

  1. Version published to 10.1186/s12916-020-01872-8
  2. NCRC

    Our take

    This study, available as a preprint and thus not yet peer reviewed, evaluated the impact of 13 different non-pharmaceutical interventions on the time varying reproductive number across 130 countries and territories through June 2020. Impact was assessed using a wide range of models with different time lags (i.e., time frame from implementation of intervention to its impact on transmission) and at varying levels of intensity of the interventions. School closures and internal movement restrictions showed evidence of impact across all models, and thus had strong evidence of effectiveness. However, these results should be interpreted cautiously as multiple interventions were often implemented at the same time, making it difficult to fully disentangle the impact of any single intervention from the others.

    Study design

    ecological;modeling-simulation

    Study population and setting

    This study used the implementation dates of a large variety of non-pharmaceutical policy interventions and assed their relationship with COVID-19 time varying reproductive number, Rt, defined as the mean number of secondary cases that one index case will infect at time t) across 130 countries and territories between January 1 and June 22, 2020. Data on non-pharmaceutical COVID-19 policies, categories, implementation dates, and a general index of strength of COVID-19 policy response were obtained from the Oxford COVID-19 Government Response Tracker. Estimates of the Rt across regions and time were from EpiForecasts. There were four primary analyses. The first main analysis used a statistical regression model to characterize how different types of NPIs were rolled out over time, such as when they occurred and in what order. The second model observed how those NPIs were clustered in time (i.e. what policies were more likely to occur together with what other policies, forming a group or a cluster of policies). A third model attempted to determine the time lags between policy implementation and their effect on Rt by assessing 3 different time lags (1, 5, and 10 days) and estimating goodness-of-fit statistics for each model. Finally, the study used panel regression models to attempt to disentangle the impact of different types of NPIs on Rt accounting for different time lags and the effort at which NPI was implemented (any effort and maximum effort).

    Summary of main findings

    First, the study finds a major increase in NPI intensity across the world in mid-March, followed by a slow reduction in the stringency of interventions. Second, the study finds substantial evidence that NPI policies were more likely to be rolled out in particular orders, typically clustered together in time, depending in part on how policy intensity is defined (any effort or maximum effort). Thirdly, the study found support for lag times between policy implementation and Rt impact between 1-10 days. Finally, the study finds evidence that school closure and internal movement restrictions, and high-intensity public events cancellations and restrictions on gatherings. There was some evidence for impact for workplace closure, income support and debt/contract relief. Evidence for impact was inconclusive for stay-at-home requirements, public information campaigns, public transport closure, international travel controls, testing, and contact tracing.

    Study strengths

    This study critically examines and demonstrates how NPI policies are related to each other, and assesses lags in their impact. These are issues that are critical both to this analysis, and many other policy analyses which often do not acknowledge that NPI policies are highly correlated with respect to timing of implementation, and that interventions have lagging effects, due to changing compliance over time. The study uses well-vetted data appropriate for policy impact evaluation. The discussion section contains a frank and well-written interpretation of the results and the interpretable limitations thereof. We find that the findings that NPIs overall had substantial impact on Rt to be relatively robust.

    Limitations

    While the discussion section involves cautious interpretation of impact of individual NPI because they were implemented at similar times, the highest impact section (the abstract) does not heed that caution, and strongly implies that they identified which specific types of NPIs were most effective despite temporal correlation. The key difficulty - one examined and partially established in the paper itself - is that these interventions are temporally related to each other, and also have time lagged effects in similar timescales as the policy rollouts. One major issue is that there appears to be substantial limitations in the way in which lagged effects were measured. Lags were assumed to be at a fixed amount of time and the same for all policies, but in reality policy lag effects are spread out over a wide period of time, and will be different for different policies in different situations. This, in addition to the close clustering in time between policies, makes it difficult to conclude which NPIs had the strongest evidence for impact.

    Value added

    The study shows strong evidence that NPI policies are related to each other over time and how that creates difficulty in examining their impact.

  3. ScreenIT

    SciScore for 10.1101/2020.08.11.20172643: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: Thank you for sharing your code.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Nonetheless, our study also has several limitations. First, besides the information bias in the NPIs database discussed above, the coding scheme may also introduce potential bias. NPIs coded as “comprehensive contact tracing for all identified cases” may have different implications in different countries. Effectiveness of contact tracing in places like Singapore (39) may be masked by seemingly similar but realistic non-comparable contact tracing programs. Second, compared to daily incidence, Rt estimates are much more suitable to compare across countries and thus is used as the metric for COVID-19 transmission in this study. However, these estimates are based on a series of assumptions (e.g., distribution of onset to confirmation delay) that may not always be appropriate. Our current model also does not factor in uncertainty around Rt estimates. Last but not the least, although we examined a wide range of NPIs, we did not include any potential interactions in the current model. Such interaction is a possibility, e.g., more people may comply with workplace closures when receiving income support. Future research should look into these relationships.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  4. Version published to 10.1101/2020.08.11.20172643v1 on medRxiv