Forecasting the COVID-19 Pandemic: Lessons learned and future directions

  1. Saketh Sundar
  2. Patrick Schwab
  3. Jade Z.H. Tan
  4. Santiago Romero-Brufau
  5. Leo Anthony Celi
  6. Dechen Wangmo
  7. Nicolás Della Penna

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

I.

The Coronavirus Disease 2019 (COVID-19) has demonstrated that accurate forecasts of infection and mortality rates are essential for informing healthcare resource allocation, designing countermeasures, implementing public health policies, and increasing public awareness. However, there exist a multitude of modeling methodologies, and their relative performances in accurately forecasting pandemic dynamics are not currently comprehensively understood.

In this paper, we introduce the non-mechanistic MIT-LCP forecasting model, and assess and compare its performance to various mechanistic and non-mechanistic models that have been proposed for forecasting COVID-19 dynamics. We performed a comprehensive experimental evaluation which covered the time period of November 2020 to April 2021, in order to determine the relative performances of MIT-LCP and seven other forecasting models from the United States’ Centers for Disease Control and Prevention (CDC) Forecast Hub.

Our results show that there exist forecasting scenarios well-suited to both mechanistic and non-mechanistic models, with mechanistic models being particularly performant for forecasts that are further in the future when recent data may not be as informative, and non-mechanistic models being more effective with shorter prediction horizons when recent representative data is available. Improving our understanding of which forecasting approaches are more reliable, and in which forecasting scenarios, can assist effective pandemic preparation and management.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.11.06.21266007: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    The forecasts were formatted for submission to the CDC Forecast Hub and then submitted to their Github repository. D. Software and Algorithms: The real-time forecasting model was implemented using Python 3.6 in Jupyter notebooks.
    Python
    suggested: (IPython, RRID:SCR_001658)
    Python packages used include NumPy [16], Pandas [17], scikit-learn [18], and XGBoost [9].
    NumPy
    suggested: (NumPy, RRID:SCR_008633)
    scikit-learn
    suggested: (scikit-learn, RRID:SCR_002577)
    “Population in Poverty” also had a high SHAP value, as did “Total Hospitals”, which line up with findings of past studies into the levels of transmission in areas with high poverty or areas with less access to healthcare.
    Poverty”
    suggested: None

    Results from OddPub: Thank you for sharing your code.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    One major limitation of many non-mechanistic models is the lack of causal inference. Integration of causal inference methods with machine learning in non-mechanistic models could improve the overall performance of the model. A second limitation is the lack of policy data integration with forecasting models. Datasets of policies and public health guidelines issued at the state and local-level could aid in the creation and advancement of forecasting models. With these policy datasets, conditional predictions could be standardized, where forecasts are generated depending on the policies that could be implemented. This improvement could increase the accuracy of forecasts as well as expand the potential impact of the forecasts on policy decisions.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.11.06.21266007v1 on medRxiv