Methodological quality of COVID-19 clinical research

  1. Richard G. Jung
  2. Pietro Di Santo
  3. Cole Clifford
  4. Graeme Prosperi-Porta
  5. Stephanie Skanes
  6. Annie Hung
  7. Simon Parlow
  8. Sarah Visintini
  9. F. Daniel Ramirez
  10. Trevor Simard
  11. Benjamin Hibbert

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

The COVID-19 pandemic began in early 2020 with major health consequences. While a need to disseminate information to the medical community and general public was paramount, concerns have been raised regarding the scientific rigor in published reports. We performed a systematic review to evaluate the methodological quality of currently available COVID-19 studies compared to historical controls. A total of 9895 titles and abstracts were screened and 686 COVID-19 articles were included in the final analysis. Comparative analysis of COVID-19 to historical articles reveals a shorter time to acceptance (13.0[IQR, 5.0–25.0] days vs. 110.0[IQR, 71.0–156.0] days in COVID-19 and control articles, respectively; p  < 0.0001). Furthermore, methodological quality scores are lower in COVID-19 articles across all study designs. COVID-19 clinical studies have a shorter time to publication and have lower methodological quality scores than control studies in the same journal. These studies should be revisited with the emergence of stronger evidence.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.07.02.20145102: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    The searches were conducted in MEDLINE (Ovid MEDLINE(R) ALL 1946-), Embase (Ovid Embase Classic + Embase 1947-) and the Cochrane Central Register of Controlled Trials (from inception).
    MEDLINE
    suggested: (MEDLINE, RRID:SCR_002185)
    Embase
    suggested: (EMBASE, RRID:SCR_001650)
    Cochrane Central Register of Controlled Trials
    suggested: (Cochrane Central Register of Controlled Trials, RRID:SCR_006576)
    All statistical analyses were performed using SAS v9.4 (SAS Institute, Inc.
    SAS Institute
    suggested: (Statistical Analysis System, RRID:SCR_008567)
    All figures were generated using GraphPad Prism v8 (GraphPad Software, La Jolla, CA, USA).
    GraphPad Prism
    suggested: (GraphPad Prism, RRID:SCR_002798)
    GraphPad
    suggested: (GraphPad Prism, RRID:SCR_002798)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Our study is not without limitations. We evaluated the methodological quality of existing studies using established quality scores. While it is tempting to associate quality scores with reproducibility, it is not possible to ascertain the impact on the design and conduct of research nor results or conclusions in the identified reports.34 Second, our analysis includes early publications on COVID-19 and there is likely to be an improvement in quality of related studies and study design as the field matures and higher quality studies which take longer to design, conduct, and report are published. Accordingly, our findings are limited to the early body of research as it pertains to the pandemic and it is likely that over time research quality will improve.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1038/s41467-021-21220-5
  3. ScreenIT

    SciScore for 10.1101/2020.07.02.20145102: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.Randomization– Coronavirus disease 2019 NOS – Newcastle-Ottawa Scale OR – odds ratio RCT – randomized controlled trial SARS-CoV-2 - Severe acute respiratory syndrome coronavirus 2 Abstract Objective: To systematically evaluate the quality of reporting of currently available COVID-19 studies compared to historical controls .Blindingnot detected.Power Analysisnot detected.Sex as a biological variablenot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Design: A systematic review and case-control analysis Data sources: MEDLINE , Embase , and Cochrane Central Register of Controlled Trials until May 14 , 2020 Study selection: All original clinical literature evaluating COVID-19 or SARS-CoV2 were identified and 1:1 historical control of the same study type in the same published journal was matched from the previous year Data extraction: Two independent reviewers screened titles , abstracts , and full-texts and independently assessed methodological quality using Cochrane Risk of Bias Tool , NewcastleOttawa Scale , QUADAS-2 Score , or case series checklist .
    Cochrane Risk
    suggested: None
    The searches were conducted in MEDLINE ( Ovid MEDLINE(R ) ALL 1946-) , Embase ( Ovid Embase Classic + Embase 1947- ) and the Cochrane Central Register of Controlled Trials ( from inception) .
    MEDLINE
    suggested: (MEDLINE, SCR_002185)
          <div style="margin-bottom:8px">
        <div><b>Embase</b></div>
        <div>suggested: (EMBASE, <a href="https://scicrunch.org/resources/Any/search?q=SCR_001650">SCR_001650</a>)</div>
      </div>
    
      <div style="margin-bottom:8px">
        <div><b>Cochrane Central Register of Controlled Trials</b></div>
        <div>suggested: (Cochrane Central Register of Controlled Trials, <a href="https://scicrunch.org/resources/Any/search?q=SCR_006576">SCR_006576</a>)</div>
      </div>
    </td></tr><tr><td style="min-width:100px;vertical-align:top;border-bottom:1px solid lightgray">All statistical analyses were performed using SAS v9.4 ( SAS Institute , Inc .</td><td style="min-width:100px;border-bottom:1px solid lightgray">
      <div style="margin-bottom:8px">
        <div><b>SAS Institute</b></div>
        <div>suggested: (Statistical Analysis System, <a href="https://scicrunch.org/resources/Any/search?q=SCR_008567">SCR_008567</a>)</div>
      </div>
    </td></tr><tr><td style="min-width:100px;vertical-align:top;border-bottom:1px solid lightgray">All figures were generated using GraphPad Prism v8 ( GraphPad Software , La Jolla , CA , USA) .</td><td style="min-width:100px;border-bottom:1px solid lightgray">
      <div style="margin-bottom:8px">
        <div><b>GraphPad Prism</b></div>
        <div>suggested: (GraphPad Prism, <a href="https://scicrunch.org/resources/Any/search?q=SCR_002798">SCR_002798</a>)</div>
      </div>
    
      <div style="margin-bottom:8px">
        <div><b>GraphPad</b></div>
        <div>suggested: (GraphPad Prism, <a href="https://scicrunch.org/resources/Any/search?q=SCR_002798">SCR_002798</a>)</div>
      </div>
    </td></tr><tr><td style="min-width:100px;vertical-align:top;border-bottom:1px solid lightgray">Hibbert reports funding as a clinical trial investigator from Abbott , Boston Scientific , and Edwards Lifesciences outside of the submitted work .</td><td style="min-width:100px;border-bottom:1px solid lightgray">
      <div style="margin-bottom:8px">
        <div><b>Abbott</b></div>
        <div>suggested: (Abbott, <a href="https://scicrunch.org/resources/Any/search?q=SCR_010477">SCR_010477</a>)</div>
      </div>
    </td></tr></table>

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from OddPub: We did not find a statement about open data. We also did not find a statement about open code. Researchers are encouraged to share open data when possible (see Nature blog).

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore is not a substitute for expert review. SciScore checks for the presence and correctness of RRIDs (research resource identifiers) in the manuscript, and detects sentences that appear to be missing RRIDs. SciScore also checks to make sure that rigor criteria are addressed by authors. It does this by detecting sentences that discuss criteria such as blinding or power analysis. SciScore does not guarantee that the rigor criteria that it detects are appropriate for the particular study. Instead it assists authors, editors, and reviewers by drawing attention to sections of the manuscript that contain or should contain various rigor criteria and key resources. For details on the results shown here, including references cited, please follow this link.

  4. Version published to 10.1101/2020.07.02.20145102v1 on medRxiv