Defining the analytical and clinical sensitivity of the ARTIC method for the detection of SARS-CoV-2

  1. Nabil-Fareed Alikhan
  2. Joshua Quick
  3. Alexander J. Trotter
  4. Samuel C. Robson
  5. Matthew Bashton
  6. Gemma L. Kay
  7. Matt Loose
  8. Stefan Rooke
  9. Martin McHugh
  10. Alistair C Darby
  11. Samuel M. Nicholls
  12. Nicholas J. Loman
  13. The COVID-19 Genomics UK (COG-UK) consortium
  14. Samir Dervisevic
  15. Andrew J. Page
  16. Justin O’Grady

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

The SARS-CoV-2 ARTIC amplicon protocol is the most widely used genome sequencing method for SARS-CoV-2, accounting for over 43% of publicly-available genome sequences. The protocol utilises 98 primers to amplify ∼400bp fragments of the SARS-CoV-2 genome covering all 30,000 bases. Understanding the analytical performance metrics of this protocol will improve how the data is used and interpreted. Different concentrations of SARS-CoV-2 control material were used to establish the limit of detection (LoD) of the ARTIC protocol. Results demonstrated the LoD was a minimum of 25-50 virus particles per mL. The sensitivity of ARTIC was comparable to the published sensitivities of commercial diagnostics assays and could therefore be used to confirm diagnostic testing results. A set of over 3,600 clinical samples from three UK regions were then evaluated to compare the protocols performance to clinical diagnostic assays (Roche Lightcycler 480 II, AusDiagnostics, Roche Cobas, Hologic Panther, Corman RdRp, Roche Flow, ABI QuantStudio 5, Seegene Nimbus, Qiagen Rotorgene, Abbott M2000, Thermo TaqPath, Xpert). We developed a Python tool, RonaLDO, to perform this validation (available under the GNU GPL3 open-source licence from https://github.com/quadram-institute-bioscience/ronaldo ). Positives detected by diagnostic platforms were generally supported by sequencing data; platforms that used RT-qPCR were the best predictors of whether the sample would subsequently sequence successfully. To maximise success of sample sequencing for phylogenetic analysis, samples with Ct <31 should be chosen. For diagnostic tests that do not provide a quantifiable Ct value, adding a quantification step is recommended. The ARTIC SARS-CoV-2 sequencing protocol is highly sensitive, capable of detecting SARS-CoV-2 in samples with Cts in the high 30s. However, to routinely obtain whole genome coverage, samples with Ct <31 are recommended. Comparing different virus detection methods close to their LoD was challenging and significant discordance was observed.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.10.09.21264695: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Ethicsnot detected.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Sequenced read data from the PORT site, which were all generated through the Oxford Nanopore platform, were demultiplexed using MinKNOW with ‘--require_barcodes_both_ends’ set.
    MinKNOW
    suggested: None
    Data release: Raw sequence data from clinical samples were deposited in and are available from the European Nucleotide Archive under BioProject accession number PRJEB37886.
    BioProject
    suggested: (NCBI BioProject, RRID:SCR_004801)
    RonaLDO: RonaLDO is Python version 3 software created as part of this study to validate the sequence data and is now available under the GNU GPL3 open source licence from: https://github.com/quadram-institute-bioscience/ronaldo.
    Python
    suggested: (IPython, RRID:SCR_001658)

    Results from OddPub: Thank you for sharing your code and data.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    It is thus important to understand the limitations of the ARTIC protocol and how sequencing data compares with the results provided by diagnostic platforms when both are applied to the same samples. Here we define the analytical and clinical sensitivity of ARTIC for the detection of SARS-CoV-2 in mock and clinical samples respectively. We also define the RT-qPCR Ct thresholds that provide high quality sequencing results (>80% and 90% genome recovery) so that only positive samples likely to provide useful information can be sequenced. By processing SARS-CoV-2 samples of known concentrations, in triplicate, we established the LoD for the ARTIC protocol as 25-50 viral copies per mL (Figure 1). These values can be compared with listed sensitivities of diagnostic tests, and showed ARTIC was comparable, if not slightly more sensitive than the platforms described (Table 1). However, genome recovery (% of genome covered at >10x) at the LoD is <10% (Fig 1A) and unlikely to provide useful lineage information. Genome recovery increases to an average of 35% when 100 viral copies are present in the ARTIC PCR which equates to approx Ct 31 (for a sensitive RT-qPCR assay). Therefore it could be estimated that positive clinical samples with Ct <31 would be required to provide >50% genome recovery using ARTIC, assuming the diagnostic assays are very sensitive and the volume of RNA extract tested in the RT-qPCR is similar to ARTIC (11ul). Sequenced reads from known concentrations of SARS-CoV-2 ...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.10.09.21264695v1 on medRxiv