Use of Modeling to Inform Decision Making in North Carolina during the COVID-19 Pandemic: A Qualitative Study

  1. Karl Johnson
  2. Caitlin B. Biddell
  3. Kristen Hassmiller Lich
  4. Julie Swann
  5. Paul Delamater
  6. Maria Mayorga
  7. Julie Ivy
  8. Raymond L. Smith
  9. Mehul D. Patel

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Background. The COVID-19 pandemic has popularized computer-based decision-support models, which are commonly used to inform decision making amidst complexity. Understanding what organizational decision makers prefer from these models is needed to inform model development during this and future crises. Methods. We recruited and interviewed decision makers from North Carolina across 9 sectors to understand organizational decision-making processes during the first year of the COVID-19 pandemic ( N = 44). For this study, we identified and analyzed a subset of responses from interviewees ( n = 19) who reported using modeling to inform decision making. We used conventional content analysis to analyze themes from this convenience sample with respect to the source of models and their applications, the value of modeling and recommended applications, and hesitancies toward the use of models. Results. Models were used to compare trends in disease spread across localities, estimate the effects of social distancing policies, and allocate scarce resources, with some interviewees depending on multiple models. Decision makers desired more granular models, capable of projecting disease spread within subpopulations and estimating where local outbreaks could occur, and incorporating a broad set of outcomes, such as social well-being. Hesitancies to the use of modeling included doubts that models could reflect nuances of human behavior, concerns about the quality of data used in models, and the limited amount of modeling specific to the local context. Conclusions. Decision makers perceived modeling as valuable for informing organizational decisions yet described varied ability and willingness to use models for this purpose. These data present an opportunity to educate organizational decision makers on the merits of decision-support modeling and to inform modeling teams on how to build more responsive models that address the needs of organizational decision makers.

Organizations from a diversity of sectors across North Carolina (including public health, education, business, government, religion, and public safety) have used decision-support modeling to inform decision making during COVID-19. Decision makers wish for models to project the spread of disease, especially at the local level (e.g., individual cities and counties), and to help estimate the outcomes of policies. Some organizational decision makers are hesitant to use modeling to inform their decisions, stemming from doubts that models could reflect nuances of human behavior, concerns about the accuracy and precision of data used in models, and the limited amount of modeling available at the local level.

Article activity feed

  1. Version published to 10.1177/23814683221116362
  2. ScreenIT

    SciScore for 10.1101/2021.04.13.21255401: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementIRB: This study was exempted from IRB review by the [blinded-for-review] Office of Human Research Ethics.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.04.13.21255401v1 on medRxiv