On the sensitivity of non-pharmaceutical intervention models for SARS-CoV-2 spread estimation
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
Introduction
A series of modelling reports that quantify the effect of non-pharmaceutical interventions (NPIs) on the spread of the SARS-CoV-2 virus have been made available prior to external scientific peer-review. The aim of this study was to investigate the method used by the Imperial College COVID-19 Research Team (ICCRT) for estimation of NPI effects from the system theoretical viewpoint of model identifiability.
Methods
An input-sensitivity analysis was performed by running the original software code of the systems model that was devised to estimate the impact of NPIs on the reproduction number of the SARS-CoV-2 infection and presented online by ICCRT in Report 13 on March 30 2020. An empirical investigation was complemented by an analysis of practical parameter identifiability, using an estimation theoretical framework.
Results
Despite being simplistic with few free parameters, the system model was found to suffer from severe input sensitivities. Our analysis indicated that the model lacks practical parameter identifiability from data. The analysis also showed that this limitation is fundamental, and not something readily resolved should the model be driven with data of higher reliability.
Discussion
Reports based on system models have been instrumental to policymaking during the SARS-CoV-2 pandemic. With much at stake during all phases of a pandemic, we conclude that it is crucial to thoroughly scrutinise any SARS-CoV-2 effect analysis or prediction model prior to considering its use as decision support in policymaking. The enclosed example illustrates what such a review might reveal.
Article activity feed
-
SciScore for 10.1101/2020.06.10.20127324: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your code and data.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:We found the combined effects of high input sensitivity, and the assumption of R(t) being driven solely by the NPIs, to constitute a fundamental limitation which should be considered when the modelling results are used as a basis for policymaking. It is hard to judge whether, for example, a partial transition to online teaching constitutes a school closure or not. Similarly, the crowd size limit associated with the public events ban …
SciScore for 10.1101/2020.06.10.20127324: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your code and data.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:We found the combined effects of high input sensitivity, and the assumption of R(t) being driven solely by the NPIs, to constitute a fundamental limitation which should be considered when the modelling results are used as a basis for policymaking. It is hard to judge whether, for example, a partial transition to online teaching constitutes a school closure or not. Similarly, the crowd size limit associated with the public events ban NPI remains a free design parameter for the modeller to decide. The point is not to argue if a school closure took place or not, or what the most appropriate crowd size limit is. Instead, our findings highlight remarkably large effects resulting from minor changes in input data. If the NPI modelling results are to be used as support for policy decisions, it is not acceptable that subtle interpretations of NPI definitions in a single small country, like those reported here, have a pivotal impact on the estimated intervention effects in all 14 modelled countries. It is hence inherently difficult to confidently ascribe changes in Rt to specific NPIs that jointly took place over a matter of days, based on data with inconsistent detection and reporting of cases. Since the effect of each NPI is modelled by a multiplicative factor applied to Rt, an NPI that is deemed 75 % effective reduces Rt by a multiplicative factor of 1-0.75=0.25. A country that effectuates two NPIs with 50 % effectiveness on the same day will, therefore, see exactly the same reducti...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
