Performance and health economic evaluation of the Mount Sinai COVID-19 serological assay identifies modification of thresholding as necessary to maximise specificity of the assay

  1. Stuart A Rushworth
  2. Benjamin B Johnson
  3. Karen Ashurst
  4. Rose Davidson
  5. Paige Paddy
  6. Jayna J Mistry
  7. Jamie A Moore
  8. Charlotte Hellmich
  9. Dylan R Edwards
  10. Daniela Da Luz Afonso
  11. Georgios Xydopoulos
  12. Richard Goodwin
  13. Javier Gomez
  14. Reenesh Prakash
  15. Samir Dervisevic
  16. Richard Fordham
  17. Kristian M Bowles
  18. James G W Smith

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

We evaluated the FDA approved SARS-CoV-2 immunoassay (developed at Mount Sinai, by Krammer and colleagues) for the identification of COVID-19 seroconversion and potential cross-reactivity of the assay in a United Kingdom (UK) National Health Service (NHS) hospital setting. In our ‘set up’ cohort we found that the SARS-CoV-2 IgG was detectable in 100% of patients tested 14 days post positive COVID-19 nucleic acid test. Serum samples taken from pregnant women in 2018 were used as a negative control group with zero false positives. We also analysed samples from patients with non-COVID-19 viral infections, paraproteinaemia or autoantibodies and found false positive results in 6/179. Modification of the sensitivity threshold to five standard deviations from the mean of the control group eliminated all false positive result in the ‘set up’ cohort. We confirmed the validity of the test with a revised threshold on an independent prospective ‘validation cohort’ of patient samples. Taking data from both cohorts we report a sensitivity of the Mount Sinai assay of 96.6% (28/29) and specificity of 100% (299/299) using a revised threshold cut-off, at a time point at least 14 days since the diagnostic antigen test. Finally, we conducted a health economic probabilistic sensitivity analysis (PSA) on the costs of producing the tests, and the mean cost we estimate to be 13.63 pounds sterling (95%CI 9.63 - 18.40), allowing its cost effectiveness to be tested against other antibody tests. In summary, we report that the Mount Sinai IgG ELISA assay is highly sensitive test for SARS-Cov-2 infection, however modification of thresholding was required to minimise false positive results.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.06.11.20128306: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementIRB: Human serum samples: This study was sponsored by the Norfolk and Norwich University Hospital NHS Trust (UK) and conducted under ethical approval of the University of East Anglia Faculty of Medicine and Health Research Ethics Committee (UK).
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.
    Cell Line Authenticationnot detected.

    Table 2: Resources

    Antibodies
    SentencesResources
    Reactivity of plates to SARS-CoV antibody was confirmed through the use of the monoclonal antibody CR30229 (16) (Absolute Antibody) (Supplementary figure 1).
    SARS-CoV
    suggested: None
    Following a 2 hour incubation at room temperature, the ELISA plates were washed three times with PBS-T and incubated for 1 hour (± 5 minutes) with HRP-labelled anti-human Fab specific secondary IgG, IgA, or IgM antibody (Sigma), diluted 1:3000 in PBS-T containing 1% milk powder (w/v).
    anti-human Fab specific secondary IgG
    suggested: None
    IgM
    suggested: (Thermo Fisher Scientific Cat# MA1-4803, RRID:AB_612248)
    Experimental Models: Cell Lines
    SentencesResources
    Plasmid DNA was transfected into HEK293F’ cells (ThermoFisher Scientific) and Histagged protein were isolated 72 hours post transfection through gravity flow purification.
    HEK293F’
    suggested: None

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2020.06.11.20128306 on medRxiv