Can We Really Trust the Findings of the COVID-19 Research? Quality Assessment of Randomized Controlled Trials Published on COVID-19

  1. Athira S Joshy
  2. Christy Thomas
  3. Saphal Surendran
  4. Krishna Undela

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Objective

To evaluate the quality of randomized controlled trials (RCTs) published on Coronavirus Disease-19 (COVID-19) and to investigate the reasons behind compromising the quality, if found.

Methods

A systematic literature search was performed in PubMed, Google Scholar, and Cochrane CENTRAL to identify the Randomized Controlled Trails published on Coronavirus Disease-19 between 1 st Dec 2019 to 31 st Aug 2021. Research articles met with study criteria were included in the study. Assessment of quality of randomized controlled trials was done using modified Jadad scale.

Results

21,259 records of randomized controlled trials were identified through database searching, out of which 90 randomized controlled trials were included in the study and, 34 (37.8%) were of high-quality, 46 (51.1%) were of moderate quality, and 10 (11.1 %) were of low-quality studies. There were 40 (44.4%), 38 (42.2%), and 12 (13.3%) randomized controlled trials published in the early, middle, and late terms with Jadad score 5.12±1.67, 5.34±1.32, and 5.68±1.50 respectively (P=0.52). When comparing the blinding status, appropriate blinding, and methods to evaluate adverse events in randomized controlled trials with modified Jadad score, a significant difference was observed (P<0.001). A significant moderate positive correlation was found between the impact factor of the journal and the modified Jadad scale score (R2= 0.48, P<0.001).

Conclusion

Findings from our study indicate that accelerated publication of Coronavirus Disease-19 researches along with the fast-track review process has resulted in lowering study quality scores. With the emergence of stronger evidence, Coronavirus Disease-19 clinical studies with lower methodological quality should be revisited.

Impacts on practice

  • There have been numerous sacrifices and tragedies in the clinical response to covid-19. Revising the quality of randomized controlled trials published on COVID-19 as we enter the third wave of the pandemic and beyond, will improve the evidence-based practice of medications for clinical pharmacy services.

  • COVID-19 Patients will benefit from evidence-based pharmaceutical care through reduced drug-related problems.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2022.04.15.22273881: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Ethicsnot detected.
    Sex as a biological variablenot detected.
    RandomizationUsing this scale, studies were scored according to the presence of six features of RCTs, like randomization, blinding, withdrawals or dropouts, inclusion or exclusion criteria, statistical analysis, and the method used to assess the adverse events.
    BlindingUsing this scale, studies were scored according to the presence of six features of RCTs, like randomization, blinding, withdrawals or dropouts, inclusion or exclusion criteria, statistical analysis, and the method used to assess the adverse events.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Search strategy: The electronic databases such as PubMed, Google Scholar, and Cochrane CENTRAL were searched from 1st December 2019 to 31st August 2021 MeSH (Medical Subject Headings) terms were identified and were combined using Boolean operators while searching in PubMed.
    PubMed
    suggested: (PubMed, RRID:SCR_004846)
    Google Scholar
    suggested: (Google Scholar, RRID:SCR_008878)
    Cochrane CENTRAL
    suggested: (Cochrane Central Register of Controlled Trials, RRID:SCR_006576)
    The search strategy followed in PubMed for RCTs was ((“covid 19”[Supplementary Concept] OR “Coronavirus Infections”[MeSH Terms:noexp] OR “severe acute respiratory syndrome coronavirus 2”[Supplementary Concept]) AND (“randomized controlled trial”[Publication Type] OR (“controlled clinical trial”[Publication Type])).
    MeSH
    suggested: (MeSH, RRID:SCR_004750)
    Data extraction and management: Research articles that met the study criteria were included in the study and data extraction was performed using Microsoft Excel.
    Microsoft Excel
    suggested: (Microsoft Excel, RRID:SCR_016137)
    SPSS v.
    SPSS
    suggested: (SPSS, RRID:SCR_002865)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    However, that study only examined the quality of scientific publications published in prominent clinical journals, which was a major limitation [9]. Our findings backed with the findings of a review of methodological quality of COVID-19 research articles published in February 2021 by Jung et al, concluded that the initial corpus of peer-reviewed COVID-19 research literature was mostly composed of observational studies that were submitted to a faster peer-review process and had lower methodological quality standards than comparable studies [10]. In our study, when comparing the modified Jadad scale score and the number of high-quality studies with the early and mid-late stages of publication, the modified Jadad scale score and the percentage of high-quality studies both increased. Early in the COVID-19 pandemic, an urgent demand for scientific data to support clinical, social, and economic decisions resulted in a shorter time to publication and an explosion of COVID-19 research published in both regular peer-reviewed journals and preprint servers. Furthermore, more than 30,000 of the COVID-19 articles published in 2020 were preprints, accounting for 17 to 30 percent of all COVID-19 research publications, and one-tenth of all preprints this year were about COVID-19, according to Dimensions [103]. This suggests that the expedited publishing of COVID-19 research, along with the accelerated review procedure, may have resulted in lower study quality scores. We also investigated the...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a protocol registration statement.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2022.04.15.22273881v1 on medRxiv