Evaluation of the comprehensiveness, accuracy and currency of the Cochrane COVID ‐19 Study Register for supporting rapid evidence synthesis production

  1. Maria‐Inti Metzendorf
  2. Robin M. Featherstone

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

The Cochrane COVID‐19 Study Register (CCSR) is a public, continually updated database of COVID‐19 study references. The aim of this study‐based register is to support rapid and living evidence synthesis, including an evidence ecosystem of COVID‐19 research (CEOsys). In November and December 2020, we conducted an evaluation of the CCSR for CEOsys, measured its performance and identified areas for improvement. For the evaluation we generated a purposive sample of 286 studies from 20 reviews to calculate the CCSR's comprehensiveness (sensitivity), accuracy (correctly classified and linked studies) and currency (time to publish and process references). Our sample showed that the CCSR had an overall comprehensiveness of 77.2%, with the highest coverage for interventional studies (94.4%). The study register had 100% coverage for trial registry records, 86.5% for journal articles and 52.4% for preprints. A total of 98.3% of references were correctly classified with regard to study type, and 93.4% with regard to study aim. A total of 89% of studies were correctly linked. A total of 81.4% of references were published to the register in under 30 days, with 0.5 day (median) for trial registry records, 2 days for journal articles and 56 days for preprints. The CCSR had high comprehensiveness, accurate study classifications and short publishing times for journal articles and trial registry records in the sample. We identified that coverage and publishing time for preprints needed improvement. Finally, the evaluation illustrated the value of a study‐based register for identifying additional study references for analysis in evidence synthesis.

Article activity feed

  1. Version published to 10.1002/jrsm.1501
  2. ScreenIT

    SciScore for 10.1101/2021.04.21.21255874: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    EthicsField Sample Permit: Data collection was carried out as planned between 21 November and 16 December 2020.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Inclusion criteria: Exclusion criteria: A PubMed search (Supplementary file 1) was used to generate reviews for the sample.
    PubMed
    suggested: (PubMed, RRID:SCR_004846)
    All evaluation study data were recorded in Excel and are publicly available.
    Excel
    suggested: None

    Results from OddPub: Thank you for sharing your data.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    4.1 Limitations: This evaluation aimed to establish a representative sample set based on 20 reviews. This aim was achieved with regard to review type (five rapid, six living, and nine systematic reviews) and three of the six CEOsys topic areas (nine reviews on “outpatient and inpatient care”, four reviews on “diagnosis”, three reviews on “intensive and palliative care”). For the other three CEOsys topic areas, the sample selection only identified two reviews on “public health” and one each for “hospital hygiene” and “mental health”. This limitation is due to the low publication output in these fields during the period of the sample selection (July to November 2020). Because the validity for these three topic areas was limited, this evaluation can only draw conclusions on topic areas where the sample was deemed to be representative. For the purposes of evaluating the currency of the CCSR’s production processes, a manual adjustment was made to the “date available” values for references published before the CCSR was launched on 1 April 2020. Any publication dates for references that predated the launch-date of the CCSR were adjusted to 1 April 2020. This ensured our currency measurements reflected true processing times and were not inflated to account for months of processing that could not have occurred prior to the CCSR’s inception. Owing to the variability in searching methods in the reviews constituting the sample, our resource limitations and our aim to compare the results ...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.04.21.21255874 on medRxiv