When Do We Need Massive Computations to Perform Detailed COVID‐19 Simulations?

  1. Christopher B. Lutz
  2. Philippe J. Giabbanelli

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

The COVID‐19 pandemic has infected over 250 million people worldwide and killed more than 5 million as of November 2021. Many intervention strategies are utilized (e.g., masks, social distancing, vaccinations), but officials making decisions have a limited time to act. Computer simulations can aid them by predicting future disease outcomes, but they also require significant processing power or time. It is examined whether a machine learning model can be trained on a small subset of simulation runs to inexpensively predict future disease trajectories resembling the original simulation results. Using four previously published agent‐based models (ABMs) for COVID‐19, a decision tree regression for each ABM is built and its predictions are compared to the corresponding ABM. Accurate machine learning meta‐models are generated from ABMs without strong interventions (e.g., vaccines, lockdowns) using small amounts of simulation data: the root‐mean‐square error (RMSE) with 25% of the data is close to the RMSE for the full dataset (0.15 vs 0.14 in one model; 0.07 vs 0.06 in another). However, meta‐models for ABMs employing strong interventions require much more training data (at least 60%) to achieve a similar accuracy. In conclusion, machine learning meta‐models can be used in some scenarios to assist in faster decision‐making.

Article activity feed

  1. Version published to 10.1002/adts.202100343
  2. ScreenIT

    SciScore for 10.1101/2021.08.26.21262694: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    3.2 Normalization & Model Training: To perform our regressions, we used TreeLearner regressors from the Orange data mining library (orange3 version 3.26.0) running on Python (version 3.8.3).
    Python
    suggested: (IPython, RRID:SCR_001658)

    Results from OddPub: Thank you for sharing your data.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    There are three main limitations to our work. First, we focused on peer-reviewed COVID-19 ABMs that we could run to obtain data and for which we could verify our use of the model vis-a-vis published results. As shown in previous calls for transparency in COVID-19 modeling, modelers do not systematically provide their code [101]. In a transparency assessment, Jalali and colleagues found that most models do not share their code [102], which echoes similar observations about practices in Agent-Based Modeling across application domains [103, 104]. Our criteria thus meant that we could only assess a subset of existing models and it is possibly that different trends or initial error levels are observed in other models. Second, several key parameters and assumptions regarding COVID-19 continue to change. For example, a study on almost 100,000 volunteers in July 2021 found a vaccine effectiveness of 49%, which is less than the lowest value assumed in some of the previous modeling studies [12, 105]. Other studies have shown that vaccinated individuals have a comparable viral load to unvaccinated ones within the first few days [106, 107], whereas the flow diagrams used in many models have considered that vaccinated individuals were fully removed from the population. Our conclusions are limited to COVID-19 ABMs developed so far, since future models may exhibit markedly different dynamics. In particular, bifurcation may be present in future models, which would require an analysis of mode...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • No funding statement was detected.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.08.26.21262694v1 on medRxiv