Meta-analysis of the robustness of COVID-19 diagnostic kit performance during the early pandemic

  1. Chandrakumar Shanmugam
  2. Michael Behring
  3. Vishwas Luthra
  4. Sixto M Leal
  5. Sooryanarayana Varambally
  6. George J Netto
  7. Upender Manne

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Accurate detection of SARS-CoV-2 is necessary to mitigate the COVID-19 pandemic. However, the test reagents and assay platforms are varied and may not be sufficiently robust to diagnose COVID-19.

Methods

We reviewed 85 studies (21 530 patients), published from five regions of the world, to highlight issues involved in the diagnosis of COVID-19 in the early phase of the pandemic. All relevant articles, published up to 31 May 2020, in PubMed, BioRiXv, MedRiXv and Google Scholar, were included. We evaluated the qualitative (9749 patients) and quantitative (10 355 patients) performance of RT-PCR and serologic diagnostic tests for real-world samples, and assessed the concordance (5538 patients) between test performance in meta-analyses. Synthesis of results was done using random effects modelling and bias was evaluated according to QUADAS-2 guidelines.

Results

The RT-PCR tests exhibited heterogeneity in the primers and reagents used. Of 1957 positive RT-PCR COVID-19 participants, 1585 had positive serum antibody (IgM±IgG) tests (sensitivity 0.81, 95% CI 0.66 to 0.90). While 3509 of 3581 participants RT-PCR negative for COVID-19 were found negative by serology testing (specificity 0.98, 95% CI 0.94 to 0.99). The chemiluminescent immunoassay exhibited the highest sensitivity, followed by ELISA and lateral flow immunoassays. Serology tests had higher sensitivity and specificity for laboratory approval than for real-world reporting data.

Discussion

The robustness of the assays/platforms is influenced by variability in sampling and reagents. Serological testing complements and may minimise false negative RT-PCR results. Lack of standardised assay protocols in the early phase of pandemic might have contributed to the spread of COVID-19.

Article activity feed

  1. Version published to 10.1136/bmjopen-2021-053912
  2. ScreenIT

    SciScore for 10.1101/2021.01.16.21249937: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Antibodies
    SentencesResources
    Variables abstracted were study authors/test developer, name of test, test platform, and true positives, false negatives, true negatives, and false positives for each antibody or antibody combination measured (IgM, IgG, IgA, combined, and Pan-Ig).
    IgM, IgG
    suggested: None
    Software and Algorithms
    SentencesResources
    Literature Search: This research was accomplished according to standards outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.15 To find relevant studies, international databases, including PubMed, MedRiXv, BioRiXv, and Google Scholar, were searched for articles published until May 31, 2020.
    PubMed
    suggested: (PubMed, RRID:SCR_004846)
    Google Scholar
    suggested: (Google Scholar, RRID:SCR_008878)
    The following search terms were used (selected using English MeSH keywords and Emtree terms): [SARS-CoV-2 AND
    MeSH
    suggested: (MeSH, RRID:SCR_004750)

    Results from OddPub: Thank you for sharing your data.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    This research has limitations. Due to the dynamic reporting of COVID-19 testing research and inconsistencies in reporting of predictive variables across studies, bias in sampling may have some effect on our results. Patient flow analysis suggests that lack of consistent RT-PCR reference standard given to patients in the same study, as well as the unclear reporting of patient selection methods could contribute to bias in these results (Fig. S2). In addition, the observed heterogeneity between studies in the meta-analysis suggests that we must consider the possibility that the differences in results may be due to chance. Lastly, it is questionable to compare two separate testing methods of RT-PCR and seroprevalence in sensitivity/specificity analysis. In particular, given the relationship between time since diagnosis and accuracy of serology testing, a contributor to the observed differences in performance is time. The effective containment of COVID-19 involves accurate diagnoses and isolation of SARS-CoV-2-infected persons. Robustness of the assays/platforms is determined by variability of the samples, primers, and reagents used. Serological tests alone are of value only during the latter times of infection; however, they complement RT-PCR when used in conjunction and minimize false negative RT-PCR results. Additionally, some of the approved serological assays/platforms, particularly those developed using contrived/laboratory data, perform poorly when applied to real-world sam...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.01.16.21249937v1 on medRxiv