Predicting the unpredictable: how dynamic COVID-19 policies and restrictions challenge model forecasts
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
Introduction
To retrospectively assess the accuracy of a mathematical modelling study that projected the rate of COVID-19 diagnoses for 72 locations worldwide in 2021, and to identify predictors of model accuracy.
Methods
Between June and August 2020, an agent-based model was used to project rates of COVID-19 infection incidence and cases diagnosed as positive from 15 September to 31 October 2020 for 72 geographic settings. Five scenarios were modelled: a baseline scenario where no future changes were made to existing restrictions, and four scenarios representing small or moderate changes in restrictions at two intervals. Post hoc, upper and lower bounds for number of diagnosed Covid-19 cases were compared with actual data collected during the prediction window. A regression analysis with 17 covariates was performed to determine correlates of accurate projections.
Results
The actual data fell within the lower and upper bounds in 27 settings and out of bounds in 45 settings. The only statistically significant predictor of actual data within the predicted bounds was correct assumptions about future policy changes (OR = 15.04; 95%CI 2.20-208.70; p=0.016).
Conclusions
For this study, the accuracy of COVID-19 model projections was dependent on whether assumptions about future policies are correct. Frequent changes in restrictions implemented by governments, which the modelling team was not always able to predict, in part explains why the majority of model projections were inaccurate compared with actual outcomes and supports revision of projections when policies are changed as well as the importance of policy experts collaborating on modelling projects.
Article activity feed
-
SciScore for 10.1101/2021.09.30.21264273: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Software and Algorithms Sentences Resources The models were generated with a population of 100,000 agents representing individuals who interact over common social layers (household, school, work, and community networks), and Covasim applies a dynamic scaling factor to the results to quantify the extent of the epidemic with respect to the size of the population in each setting. Covasimsuggested: NoneResults from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences …SciScore for 10.1101/2021.09.30.21264273: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Software and Algorithms Sentences Resources The models were generated with a population of 100,000 agents representing individuals who interact over common social layers (household, school, work, and community networks), and Covasim applies a dynamic scaling factor to the results to quantify the extent of the epidemic with respect to the size of the population in each setting. Covasimsuggested: NoneResults from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:More recently, James et al. [14] wrote of the key limitations of mathematical modelling as a tool for policymaking in the context of the global health crisis wrought by COVID-19. More specifically, describing how the rapidly changing epidemiological situation coupled with the fact that the effectiveness of new policies may not be well documented poses challenges in the evaluation and implementation of new policies in COVID-19 models. Indeed, since the start of the pandemic, COVID-19 predictions have differed significantly between the different epidemiological models and have received criticism for being wrong [15, 16]. For instance, in the initial stages of the epidemic in the United States, the number of deaths from COVID-19 on any single day fell outside the predicted range of some prominent models up to 70% of the time [17]. Eleven models assembled by FiveThirtyEight show how estimates can vary widely due to the different underlying assumptions made about policy changes and the number of contacts in the models [18]. The Centers for Disease Control and Prevention in the United States has compiled a list of 37 independent models in its published weekly forecasts predicting national and state numbers of new COVID-19 cases, deaths, and hospitalizations in the United States [19]. Moreover, these model outputs were compared against each other, validated against empirical data, and aggregated into an “ensemble” forecast to improve the predictive performance. In this study we revi...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
Results from scite Reference Check: We found no unreliable references.
-
