Reproduction as a Means of Evaluating Policy Models: A Case Study of a COVID-19 Simulation

  1. E. Chattoe-Brown
  2. N. Gilbert
  3. D. A. Robertson
  4. C. Watts

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

This article proposes (and demonstrates the effectiveness of) a new strategy for assessing the results of epidemic models which we designate reproduction . The strategy is to build an independent model that uses (as far as possible) only the published information about the model to be assessed. In the example presented here, the independent model also follows a different modelling approach (agent-based modelling) to the model being assessed (the London School of Hygiene and Tropical Medicine compartmental model which has been influential in COVID lockdown policy). The argument runs that if the policy prescriptions of the two models match then this independently supports them (and reduces the chance that they are artefacts of assumptions, modelling approach or programming bugs). If, on the other hand, they do not match then either the model being assessed is not provided with sufficient information to be relied on or (perhaps) there is something wrong with it. In addition to justifying the approach, describing the two models and demonstrating the success of the approach, the article also discusses additional benefits of the reproduction strategy independent of whether match between policy prescriptions is actually achieved.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.01.29.21250743: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.01.29.21250743 on medRxiv