The Methodological Quality Score of COVID-19 Systematic Reviews is Low, Except for Cochrane Reviews: A Meta-epidemiological Study

  1. Yuki Kataoka
  2. Shiho Oide
  3. Takashi Ariie
  4. Yasushi Tsujimoto
  5. Toshi A. Furukawa

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

No abstract available

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.08.28.20184077: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.
    RandomizationOf the articles indexed in medRxiv and PubMed and meeting the eligibility criteria, we randomly selected 100 from each database for inclusion in the present study.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    2.2 Types of studies included: We included SRs, indexed in PubMed, medRxiv, and Cochrane Database of Systematic Reviews (CDSR).
    PubMed
    suggested: (PubMed, RRID:SCR_004846)
    Cochrane Database of Systematic Reviews
    suggested: None
    We included Cochrane Reviews that dealt with the COVID-19 pandemic.
    Cochrane Reviews
    suggested: None
    1 (StataCorp LLC, College Station, Texas, United States of America).
    StataCorp
    suggested: (Stata, RRID:SCR_012763)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Referring to limitations in each abstract were more often in Cochrane than in medRxiv or PubMed. The differences of the quality of the articles we found between Cochrane reviews and others included the presentation of protocols, listing included and excluded studies, considering the quality of included studies at the conclusions, examining publication bias, and presenting the conflict of interests. These domains will be improved by the appropriate use of preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement [22,23]. If the duplicate study selection is difficult due to manpower insufficiency, one solution might be to use crowdsourcing to bring people together through the internet [24]. The proportion of protocol registrations did not differ from those reported in previous studies. Previous studies reported about 20% of SRs registered their protocols [25,26]. Protocol registration is important to prevent duplicate efforts, prevent outcome reporting bias, and to reduce alpha errors in the results of meta-analyses. PROSPERO, the largest SRs protocol registration site, has some problems including not accepting Scoping reviews and taking more than 30 days to register [27]. Some SRs published their protocols in preprint servers [28,29]. For speedy and assuring the scientific quality, protocol registration in preprint servers would be useful. Our study has several limitations. First, AMSTAR is a reliable and valid measurement tool that has been used wi...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.37737/ace.3.2_46
  3. Version published to 10.1101/2020.08.28.20184077v1 on medRxiv