Transparency and reporting characteristics of COVID-19 randomized controlled trials

  1. Philipp Kapp
  2. Laura Esmail
  3. Lina Ghosn
  4. Philippe Ravaud
  5. Isabelle Boutron

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

Background

In the context of the COVID-19 pandemic, randomized controlled trials (RCTs) are essential to support clinical decision-making. We aimed (1) to assess and compare the reporting characteristics of RCTs between preprints and peer-reviewed publications and (2) to assess whether reporting improves after the peer review process for all preprints subsequently published in peer-reviewed journals.

Methods

We searched the Cochrane COVID-19 Study Register and L·OVE COVID-19 platform to identify all reports of RCTs assessing pharmacological treatments of COVID-19, up to May 2021. We extracted indicators of transparency (e.g., trial registration, data sharing intentions) and assessed the completeness of reporting (i.e., some important CONSORT items, conflict of interest, ethical approval) using a standardized data extraction form. We also identified paired reports published in preprint and peer-reviewed publications.

Results

We identified 251 trial reports: 121 (48%) were first published in peer-reviewed journals, and 130 (52%) were first published as preprints. Transparency was poor. About half of trials were prospectively registered ( n = 140, 56%); 38% ( n = 95) made their full protocols available, and 29% ( n = 72) provided access to their statistical analysis plan report. A data sharing statement was reported in 68% ( n = 170) of the reports of which 91% stated their willingness to share. Completeness of reporting was low: only 32% ( n = 81) of trials completely defined the pre-specified primary outcome measures; 57% ( n = 143) reported the process of allocation concealment. Overall, 51% ( n = 127) adequately reported the results for the primary outcomes while only 14% ( n = 36) of trials adequately described harms. Primary outcome(s) reported in trial registries and published reports were inconsistent in 49% ( n = 104) of trials; of them, only 15% ( n = 16) disclosed outcome switching in the report. There were no major differences between preprints and peer-reviewed publications. Of the 130 RCTs published as preprints, 78 were subsequently published in a peer-reviewed journal. There was no major improvement after the journal peer review process for most items.

Conclusions

Transparency, completeness, and consistency of reporting of COVID-19 clinical trials were insufficient both in preprints and peer-reviewed publications. A comparison of paired reports published in preprint and peer-reviewed publication did not indicate major improvement.

Article activity feed

  1. Version published to 10.1186/s12916-022-02567-y
  2. ScreenIT

    SciScore for 10.1101/2022.02.03.22270357: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Ethicsnot detected.
    Sex as a biological variablenot detected.
    RandomizationThe two first pillars of this initiative are a living mapping and living evidence synthesis of all randomized controlled trials assessing treatments and preventive interventions for COVID-19.
    BlindingGeneral characteristics of the trials: We extracted the trial design, number of arms, sample size, setting, number of centers and blinding.
    Power Analysisnot detected.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Strengths and limitations: To our knowledge, this is the first study extensively assessing reporting characteristics of all COVID-19 RCTs assessing pharmacologic treatment published as preprint or peer-reviewed journal article the first 17 months of the pandemic. Our sample has been included in a large living network meta-analysis and is comprehensive. Furthermore, we assessed various dimensions of transparency, reporting and consistency between reports and registry records. Finally, to our knowledge it is the largest study comparing preprint and subsequent publications. Our study has nevertheless some limitations. First, we focused on randomized controlled trials and cannot extrapolate to other study designs. Nevertheless, RCTs are considered the gold standard for therapeutic evaluation. Second, most trials were assessed by a single researcher, however a random sample was extracted in duplicate and showed good reproducibility. Implications: Our results have important implications. There is an urgent need for high quality evidence to guide the management of COVID-19 patients. It is consequently essential to improve reporting and transparency and increase adherence to the CONSORT statement. As part of the COVID-NMA living review, we are already systematically contacting investigators to request the missing data. Further, we plan to inform investigators of their results in terms of reporting and transparency to help them improve the content of their reports. The publication of ...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2022.02.03.22270357 on medRxiv