Performance Decay of Molecular Assays Near the Limit of Detection: Probabilistic Modeling using Real-World COVID-19 Data

  1. Thomas J.S. Durant
  2. Christopher D. Koch
  3. Christopher A. Kerantzas
  4. David R. Peaper

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

The gold standard for diagnosis of COVID-19 is detection of SARS-CoV-2 RNA by RT-PCR. However, the effect of systematic changes in specimen viral burden on the overall assay performance is not quantitatively described. We observed decreased viral burdens in our testing population as the pandemic progressed, with median sample Ct values increasing from 22.7 to 32.8 from weeks 14 and 20, respectively. We developed a method using computer simulations to quantify the implications of variable SARS-CoV-2 viral burden on observed assay performance. We found that overall decreasing viral burden can have profound effects on assay detection rates. When real-world Ct values were used as source data in a bootstrap resampling simulation, the sensitivity of the same hypothetical assay decreased from 97.59 (95% CI 97.3-97.9) in week 12, to 74.42 (95% CI 73.9-75) in week 20. Furthermore, simulated assays with a 3-fold or 10-fold reduced sensitivity would both appear to be >95% sensitive early in the pandemic, but sensitivity would fall to 85.55 (95% CI 84.9-86.2) and 74.38 (95% CI 73.6-75.1) later in the pandemic, respectively. Our modeling approach can be used to better quantitate the impact that specimen viral burden may have on the clinical application of tests and specimens.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.04.26.21254638: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    EthicsField Sample Permit: It was granted Emergency Use Authorization by the US FDA, and has been previously described.(2) The preferred specimen type across all platforms was a NPS collected in 3 mL of universal or viral transport medium (VTM), but other specimen types and collection methods were used as needed based upon supply and reagent availability.
    Sex as a biological variablenot detected.
    RandomizationFrom each dataset, corresponding to an epidemiology week, individual samples and their corresponding Ct value were randomly selected using a bootstrap resampling method with replacement (Figure 1C).
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Custom scripts for data processing and analysis were implemented in Python (version 3.7.4), and R (version 4.0.2).
    Python
    suggested: (IPython, RRID:SCR_001658)
    The data from the Roche and Perkin Elmer assays were presented by RT-PCR target, while the data for the Abbott and Cepheid assays were presented for the total assay.
    Abbott
    suggested: (Abbott, RRID:SCR_010477)
    The %-Detected was plotted versus the %-LOD, and non-linear regression analysis with an exponential plateau function (where; Ymin = 0 and Ymax = 100) was fit in GraphPad Prism.
    GraphPad Prism
    suggested: (GraphPad Prism, RRID:SCR_002798)
    Additionally, the fraction detected was used to determine a probit value, probit versus Log(10) %-LOD was plotted, and linear regression was performed (GraphPad Prism v8).
    GraphPad
    suggested: (GraphPad Prism, RRID:SCR_002798)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.04.26.21254638v1 on medRxiv