Evaluation of commercially available high-throughput SARS-CoV-2 serological assays for serosurveillance and related applications

  1. Mars Stone
  2. Eduard Grebe
  3. Hasan Sulaeman
  4. Clara Di Germanio
  5. Honey Dave
  6. Kathleen Kelly
  7. Brad Biggerstaff
  8. Bridgit O. Crews
  9. Nam Tran
  10. Keith R. Jerome
  11. Thomas N. Denny
  12. Boris Hogema
  13. Mark Destree
  14. Jefferson M. Jones
  15. Natalie Thornburg
  16. Graham Simmons
  17. Mel Krajden
  18. Steve Kleinman
  19. Larry J. Dumont
  20. Michael P. Busch

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

SARS-CoV-2 serosurveys can estimate cumulative incidence for monitoring epidemics but require characterization of employed serological assays performance to inform testing algorithm development and interpretation of results. We conducted a multi-laboratory evaluation of 21 commercial high-throughput SARS-CoV-2 serological assays using blinded panels of 1,000 highly-characterized blood-donor specimens. Assays demonstrated a range of sensitivities (96%-63%), specificities (99%-96%) and precision (IIC 0.55-0.99). Durability of antibody detection in longitudinal samples was dependent on assay format and immunoglobulin target, with anti-spike, direct, or total Ig assays demonstrating more stable, or increasing reactivity over time than anti-nucleocapsid, indirect, or IgG assays. Assays with high sensitivity, specificity and durable antibody detection are ideal for serosurveillance. Less sensitive assays demonstrating waning reactivity are appropriate for other applications, including characterizing antibody responses after infection and vaccination, and detection of anamnestic boosting by reinfections and vaccine breakthrough infections. Assay performance must be evaluated in the context of the intended use.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.09.04.21262414: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    EthicsField Sample Permit: Specimens were shipped from collection sites and stored frozen at Vitalant Research Institute (VRI) until panel assembly and frozen distribution.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    BlindingUniquely blinded identical panels were distributed to experienced testing laboratories to determine performance characteristics including sensitivity, specificity, repeatability, dilutional performance, and durability of reactivity over time.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Statistical Analysis: All statistical analyses were performed using the R statistical programming language (v. 4.0.4, [6]) and using various packages, including the binom package for confidence intervals on proportions [7], the glm2 package [8] for regression analysis and the ggplot2 package [9] for plotting.
    ggplot2
    suggested: (ggplot2, RRID:SCR_014601)
    In the case of the Bio-Rad BioPlex assay (Bio-Rad, Hercules, CA), on-board dilutions were conducted by the testing lab and used to estimate reactivity in specimens where initial results were above the assay’s limit of quantitation.
    BioPlex
    suggested: (BioPlex, RRID:SCR_016144)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    The Wantai assay has been widely used in serosurveillance globally [25-27]; while this demonstrated lower specificity and reproducibility than the best performing assays, it performs adequately for serosurveillance with accounting for those limitations. Several other assays, including the Abbott IgG anti-N and EUROIMMUN IgG anti-S assays, have been employed in large-scale serosurveillance, but require adjustments for rapid waning and seroreversion to estimate cumulative incidence or attack rates, especially over longer periods and multiple epidemic waves. This study provides critical data that can be applied to adjust for waning in other studies. This study has several limitations. Asymptomatic cases are underrepresented in the panel as CCP donors qualify based on recovery from symptomatic infection, potentially resulting in overestimation of sensitivity. The assessment of durability of bAb detection is based on CCP donations from donors whose continued qualification required ongoing Ortho VITROS Total Ig anti-S1 reactivity. Although these CCP donors do not have documented dates of NAT-positivity, symptom onset or resolution, the first donations were generally within 1-2 months of symptom resolution [5]. To address these limitations we developed approaches to adequately characterize sensitivity and durability of reactivity. The number of specimens included in the dilutional series subpanels are not sufficient for robust assessment of endpoint dilutional sensitivity. This stud...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.09.04.21262414v1 on medRxiv