Performance of SARS-CoV-2 serology tests: Are they good enough?

  1. Isabelle Piec
  2. Emma English
  3. Mary Annette Thomas
  4. Samir Dervisevic
  5. William D. Fraser
  6. William Garry John

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

In the emergency of the SARS-CoV-2 pandemic, great efforts were made to quickly provide serology testing to the medical community however, these methods have been introduced into clinical practice without the complete validation usually required by the regulatory organizations. SARS-CoV-2 patient samples (n = 43) were analyzed alongside pre-pandemic control specimen (n = 50), confirmed respiratory infections (n = 50), inflammatory polyarthritis (n = 22) and positive for thyroid stimulating immunoglobulin (n = 30). Imprecision, diagnostic sensitivity and specificity and concordance were evaluated on IgG serologic assays from EuroImmun, Epitope Diagnostics (EDI), Abbott Diagnostics and DiaSorin and a rapid IgG/IgM test from Healgen. EDI and EuroImmun imprecision was 0.02–14.0% CV. Abbott and DiaSorin imprecision (CV) ranged from 5.2%–8.1% and 8.2%–9.6% respectively. Diagnostic sensitivity of the assays was 100% (CI: 80–100%) for Abbott, EDI and EuroImmun and 95% (CI: 73–100%) for DiaSorin at ≥14 days post PCR. Only the Abbott assay had a diagnostic specificity of 100% (CI: 91–100%). EuroImmun cross-reacted in 3 non-SARS-CoV-2 respiratory infections and 2 controls. The DiaSorin displayed more false negative results and cross-reacted in six cases across all conditions tested. EDI had one cross-reactive sample. The Healgen rapid test showed excellent sensitivity and specificity. Overall, concordance of the assays ranged from 76.1% to 97.9%. Serological tests for SARS-CoV-2 showed good analytical performance. The head-to-head analysis of samples revealed differences in results that may be linked to the use of nucleocapsid or spike proteins. The point of care device tested demonstrated adequate performance for antibody detection.

Article activity feed

  1. Version published to 10.1371/journal.pone.0245914
  2. ScreenIT

    SciScore for 10.1101/2020.11.13.20229625: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementIRB: Moreover, using the UK NHS Research Ethics Committee decision toolkit (http://www.hra-decisiontools.org.uk/ethics/) we confirmed that separate ethical review is not required for this study which is in concordance with the Helsinki Declaration.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Antibodies
    SentencesResources
    Pre-pandemic samples from patients who had a range of confirmed respiratory infections (including Influenza A, B and seasonal coronaviruses [Table 1]), samples collected from patients with inflammatory polyarthritis positive for anti-cyclic citrullinated peptide antibodies (anti-CCP) along with samples positive for thyroid stimulating immunoglobulin (TSI) were used to test the non-specific binding of non-SARS-CoV-2 antibodies.
    anti-cyclic citrullinated peptide antibodies
    suggested: None
    anti-CCP
    suggested: None
    A total of 195 individual serum samples (43 P, 50 N, 50 CR, 22 RA and 30 TSI) were analysed for SARS-CoV-2 IgG antibodies.
    SARS-CoV-2 IgG
    suggested: None
    Software and Algorithms
    SentencesResources
    EuroImmun and DiaSorin assays detect antibodies to, respectively, recombinant S1 and S1/S2 domains of the SARS-CoV-2 spike protein while both the EDI and Abbott assay detect antibodies to the nucleocapsid.
    Abbott
    suggested: (Abbott, RRID:SCR_010477)
    Statistics: Using IBM SPSS Statistics 25.0.0.1, Mann-Whitney and Cohen’s Kappa tests were used to compare OD results between groups and to determine the concordance between the assays, respectively.
    SPSS
    suggested: (SPSS, RRID:SCR_002865)
    Graphical representations were conducted with GraphPad Prism version 8.0 (GraphPad Software, Inc., USA).
    GraphPad Prism
    suggested: (GraphPad Prism, RRID:SCR_002798)
    GraphPad
    suggested: (GraphPad Prism, RRID:SCR_002798)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Without this knowledge the sensitivity and specificity data is brought into question and it is important that the limitations of assay are fully understood before applying the results in clinical practice. The Food and Drug Administration and European Medicines Agency acceptance criteria for biological assays typically define the required between-run and within-run precision as CV≤15 % for positive samples and ≤20 % for samples at the lower limit of quantification (10,11). All immunoassays passed the criteria for positive samples. Published median seroconversion time for IgG is around 14 days post symptoms (12–14). As we did not have access to symptom onset for most patients, we used PCR day to date the samples, before and after day 14. All samples post day 14 were positive in all assay except DiaSorin, which returned one false negative (day 39). Positivity prior to day 14 was consistent between EDI, EuroImmun and Abbott. These results are differing from those published by PHE who observed more false negative results in the Abbott than the DiaSorin (92.7% sensitivity vs 95% sensitivity, respectively) (8). We estimated seroconversion post PCR positivity to be between 9 and 12 days on these assays. Although we couldn’t do a full comparison of the POCT with the immunoassays, 100% of the P≥14 samples were IgG positive. More samples were also positive with POCT prior day 14 than in the other assays. In regard to the POCT, our study showed excellent sensitivity and specificity. We ...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a protocol registration statement.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2020.11.13.20229625v1 on medRxiv