Manufacturer Signal-to-Cutoff Threshold Underestimates Cumulative Incidence of SARS-CoV-2 Infection: Evidence from the Los Angeles Firefighters Study

  1. Omar Toubat
  2. Anders H Berg
  3. Kimia Sobhani
  4. Karen Mulligan
  5. Acacia M Hori
  6. Jay Bhattacharya
  7. Neeraj Sood

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Background

The objective of this analysis was to compare the performance sensitivity and specificity of manufacturer-recommended signal-to-cutoff (S/Co) thresholds with modified S/Co values to estimate the prevalence of SARS-CoV-2-specific antibodies in a cohort of firefighters with a known infection history.

Methods

Plasma venipuncture samples were used for serologic analysis of firefighters in Los Angeles, CA, USA, in October 2020. Seropositivity was assessed using the manufacturer’s recommended S/Co (≥1.4 IgG) and modified S/Co thresholds based on measured antibody levels in 178 negative control patients who had blood drawn prior to the emergence of COVID-19. Optimal S/Co threshold was determined by receiver operating characteristic (ROC) curve analysis.

Results

Of 585 firefighters included in the study, 52 (8.9%) reported having a PCR-positive test history prior to antibody testing. Thirty-five (67.3%) firefighters with a previous PCR-positive test were seropositive based on the manufacturer S/Co thresholds, consistent with an estimated 67.3% sensitivity and 100% specificity. After evaluating multiple modified S/Co thresholds based on pre-pandemic negative samples, a modified S/Co of 0.36 was found to yield optimal sensitivity (88.5%) and specificity (99.4%) by ROC curve analysis. This modified threshold improved serostatus classification accuracy by 21.2%.

Conclusions

S/Co thresholds based on known negative samples significantly increase seropositivity and more accurately estimate cumulative incidence of disease compared to manufacturer-based thresholds.

Article activity feed

  1. Version published to 10.1093/jalm/jfac034
  2. ScreenIT

    SciScore for 10.1101/2021.04.20.21255829: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    EthicsIRB: The Los Angeles County Department of Public Health Institutional Review Board approved this study.
    Consent: We obtained written informed consent from all study participants.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Serology testing was conducted at Cedars-Sinai Medical Center’s CLIA-certified laboratory with FDA EUA approvals, using the Abbott Architect SARS-CoV-2 assays for IgM and IgG antibodies against spike and nucleocapsid proteins (Abbott Laboratories, Chicago, IL).
    Abbott Architect
    suggested: (Abbott ARCHITECT i1000sr System, RRID:SCR_019328)
    Abbott Laboratories
    suggested: None

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    The findings of this study should be viewed in light of its limitations. First, seroprevalence was assessed based on blood specimens drawn at a single, cross-sectional time point, resulting in varying times since initial PCR positivity. Future serology studies evaluating S/Co thresholds may benefit from repeated measurements to longitudinally track antibody kinetics over time. In addition, PCR positivity was determined by participant survey, thus we cannot rule out false-positive PCR test histories. Finally, it is unclear whether factors such as age, race/ethnicity, concomitant comorbidities, and cross-reactivity with other known coronaviruses influenced SARS-CoV-2 antibody measurements in the negative control cohort utilized in this study. Additional studies evaluating SARS-CoV-2 antibody levels in larger and more representative negative control specimens are needed.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.04.20.21255829v1 on medRxiv