Reproducibility and reporting practices in COVID-19 preprint manuscripts

  1. Josh Sumner
  2. Leah Haynes
  3. Sarah Nathan
  4. Cynthia Hudson-Vitale
  5. Leslie D. McIntosh

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

The novel coronavirus, COVID-19, has sparked an outflow of scientific research seeking to understand the virus, its spread, and best practices in prevention and treatment. If this international research effort is going to be as swift and effective as possible, it will need to rely on a principle of open science. When researchers share data, code, and software and generally make their work as transparent as possible, it allows other researchers to verify and expand upon their work. Furthermore, it allows public officials to make informed decisions. In this study, we analyzed 535 preprint articles related to COVID-19 for eight transparency criteria and recorded study location and funding information. We found that individual researchers have lined up to help during this crisis, quickly tackling important public health questions, often without funding or support from outside organizations. However, most authors could improve their data sharing and scientific reporting practices. The contrast between researchers’ commitment to doing important research and their reporting practices reveals underlying weaknesses in the research community’s reporting habits, but not necessarily their science.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.03.24.20042796: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    On March 16, 2020, we harvested all preprint articles from the medRxivand bioRxiv databases related to COVID-19, for a total of 535 manuscripts.
    bioRxiv
    suggested: (bioRxiv, RRID:SCR_003933)

    Results from OddPub: Thank you for sharing your data.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    The COVID-19 pandemic has laid bare the medical systems of affected nations, exposing their strengths and weaknesses. Similarly, it has revealed patterns in the scientific community. In both the medical and research communities, COVID-19 has revealed the devoted, quality work of individuals but the many challenges of the larger systems that those individuals operate within. Since the outbreak of COVID-19 in 2019, researchers around the world have started exploring important health questions related to the virus, often without funding or outside support. Our research found that 8% of preprint articles received no funding. Furthermore, several new preprint articles related to COVID-19 are uploaded to medRxiv and bioRxiv each day. These actions reveal the dedication of scientists to provide their scientific information to decision-makers as quickly as possible. Yet, we found that most authors are not employing best reporting practices, despite the importance of open science during a health crisis. Many authors did not make their data readily accessible, did not share code, and did not engage in other important transparency practices. At a time when many scientists are going above and beyond to promote their research, these findings suggest that the scientific community may not recognize the role that certain criteria play in reproducibility and may not appreciate the extent to which open science promotes innovation and advancement. It may also signal that it is simply difficult ...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2020.03.24.20042796v1 on medRxiv