Sensitivity of ID NOW and RT–PCR for detection of SARS-CoV-2 in an ambulatory population
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
The authors evaluate the performance of the Abbott ID NOW SARS-CoV-2 test in a group of non-hospitalized individuals being tested for COVID-19 and compared that performance to an RT-PCR test. The authors also provide an interesting review and meta-analysis of ID NOW performance across the literature. The cohort assessed in this study, however, was small and localized, which currently undermines its comparison with the known performance of the test in broader applications.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 agreed to share their name with the authors.)
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Evaluated articles (ScreenIT)
Abstract
Diagnosis of SARS-CoV-2 (COVID-19) requires confirmation by reverse transcription–polymerase chain reaction (RT–PCR). Abbott ID NOW provides fast results but has been criticized for low sensitivity. Here we determine the sensitivity of ID NOW in an ambulatory population presented for testing. The study enrolled 785 symptomatic patients, of whom 21 were positive by both ID NOW and RT–PCR, and 2 only by RT–PCR. All 189 asymptomatic patients tested negative. The positive percent agreement between the ID NOW assay and the RT–PCR assay was 91.3%, and negative percent agreement was 100%. The results from the current study were included into a larger systematic review of literature where at least 20 subjects were simultaneously tested using ID NOW and RT–PCR. The overall sensitivity for ID NOW assay was calculated at 84% (95% confidence interval 55–96%) and had the highest correlation to RT–PCR at viral loads most likely to be associated with transmissible infections.
Article activity feed
-
-
Response to Public Reviews
Response to Reviewer #1 (Public Review):
We thank the reviewer for their kind comments and were glad to learn that the reviewer felt the manuscript significantly contributed to body of knowledge on COVID-19.
Response to Reviewer #2 (Public Review):
We thank the reviewer for their dedication to a detailed review, and we greatly appreciated the constructive suggestions given that have helped strengthen the overall manuscript.
We agree with the reviewer that the loss of power for detecting a loss of sensitivity by the ID NOW PCR assay was hindered by the overall low population frequency of COVID-19 disease; this resulted in a low number of positive patients and ultimately led to early study termination as a result. Because of the reviewer’s helpful observation, we have now re-estimated the power of this study …
Response to Public Reviews
Response to Reviewer #1 (Public Review):
We thank the reviewer for their kind comments and were glad to learn that the reviewer felt the manuscript significantly contributed to body of knowledge on COVID-19.
Response to Reviewer #2 (Public Review):
We thank the reviewer for their dedication to a detailed review, and we greatly appreciated the constructive suggestions given that have helped strengthen the overall manuscript.
We agree with the reviewer that the loss of power for detecting a loss of sensitivity by the ID NOW PCR assay was hindered by the overall low population frequency of COVID-19 disease; this resulted in a low number of positive patients and ultimately led to early study termination as a result. Because of the reviewer’s helpful observation, we have now re-estimated the power of this study without reference to the observed results, but with consideration for the sample size and proportion of RT-PCR positive tests that were observed when the study was terminated. This new re-estimation suggests that the study retained 80% power to find a difference of 15% or more in sensitivity between ID NOW isothermal PCR and conventional RT-PCR; this analysis also demonstrated over 95% power to find a difference in specificity of more than 5%. Indeed, the significant drop in population prevalence that led to a loss of power for detecting loss of sensitivity expectedly resulted in an increase in power for detecting loss of specificity. We have expanded the Methodssection of the paper to better expose these issues, and we have expanded our statement of strengths and limitations in the Discussion for the same reason.
The Methods section of the manuscript now reads as follows:
"The original study design called for enrolling 2000 symptomatic and 500 asymptomatic subjects, which would have provided, in the symptomatic population, power of 80 % for finding a difference (at α = 0.05) of 5% in the sensitivity of ID NOW compared with the RTPCR reference standard; inclusion of at least 1350 negative patients would have provided 95% power (at α =0.025) for finding a 5% difference in specificity. The study design assumed a population prevalence of 10%, and the study was terminated early when the population prevalence dropped to such a low level as to make the study unaffordable. We have re-estimated the power of this study without reference to the observed results but considering the sample size and proportion of RT-PCR positive tests that were observed when the study was terminated. This re-estimation suggests that the study retained 80% power to find a difference of 15% or more in sensitivity between ID NOW and RTPCR, and well over 95% power to find a difference in specificity of more than 5%. Indeed, the significant drop in population prevalence that led to a loss of power for detecting loss of sensitivity resulted, as expected (Bujang and Adnan, 2016), in an increase in power for detecting loss of specificity.
The revised section of the Discussion regarding strengths and weaknesses now reads as follows:
"Our clinical study also suffered a significant loss of power to assess ID NOW sensitivity as a result of the low number of positive results, and the reduction of sample size caused by the decision to terminate the study as a result. The meta-analysis is also limited by the small number of studies meeting inclusion criteria, and the fact that positive cases are heavily concentrated in only a single study. Strengths of the clinical study include pretrial power analysis with sample size estimation, precise adherence to the ID NOW specimen acquisition protocol, and extremely high power for assessing assay specificity. Taken together with the focus on initial diagnosis of disease in the studies included in the meta-analysis, we believe the combination of trial and meta-analysis provides useful information for clinicians for whom point-of-care testing is helpful."
We thank the reviewer for noting the unique findings from the current cohort study in comparison to existing literature. The current cohort study was done with meticulous care to identify apparent “false positives” returned by ID NOW PCR assay. This is also reflected in some of the high-quality studies available in the literature. In the supplementary data, we have provided confusion matrices for all the studies included in the meta-analysis. We have identified four such cases out of 1,942 total ID NOW tests. Cell sizes of 0 (from our study) and 4 are two small to allow use of Chi-squared for assessment of heterogeneity, and unfortunately the total number of tests is too large to allow computation of Fisher’s exact test; however, with such small numbers it is reasonable to treat them as samples drawn from Poisson distributions. The confidence interval around a Poisson estimate of 4 is 1.08987 ≤ μ ≤ 10.24159, and that around a Poisson estimate of 0 is 0.00000 ≤ μ ≤ 3.68888. The overlap is such that the estimates, 0 and 4, are consistent with having sampled the same distribution. While this does not allow us to conclude that no difference exists between our results and those of the other studies, it does not provide any evidence that there is a difference from the current cohort study and those previously published.
-
Reviewer #2 (Public Review):
Tu et al. submit a manuscript that evaluates the performance of the Abbott ID NOW SARS-CoV-2 test in an ambulatory cohort relative to RT-PCR tests. They enrolled 785 symptomatic patients, 21 tested positive for SARS-CoV-2 by ID NOW and PCR (Hologic) while 2 tested positive only via PCR. They also tested 189 asymptomatic individuals, none of whom tested positive by either ID NOW or PCR. The positive agreement between ID NOW and PCR was 91.3%, and the negative percent agreement was 100%. The authors also provide a review and meta-analysis of ID NOW performance across at least a dozen other named studies which is thorough and interesting. The cohort assessed in this study is small and localized. The data is undermined by sample size, with the most glaring example being the 100% negative percent agreement, which …
Reviewer #2 (Public Review):
Tu et al. submit a manuscript that evaluates the performance of the Abbott ID NOW SARS-CoV-2 test in an ambulatory cohort relative to RT-PCR tests. They enrolled 785 symptomatic patients, 21 tested positive for SARS-CoV-2 by ID NOW and PCR (Hologic) while 2 tested positive only via PCR. They also tested 189 asymptomatic individuals, none of whom tested positive by either ID NOW or PCR. The positive agreement between ID NOW and PCR was 91.3%, and the negative percent agreement was 100%. The authors also provide a review and meta-analysis of ID NOW performance across at least a dozen other named studies which is thorough and interesting. The cohort assessed in this study is small and localized. The data is undermined by sample size, with the most glaring example being the 100% negative percent agreement, which doesn't compare with the known performance of the test in broader populations.
-
Reviewer #1 (Public Review):
The study presents relatively high and robust sensitivity of Abbott ID NOW for the detection of SARS-CoV-2 (COVID-19) in an ambulatory population, utilizing the RT-PCR methodology as a comparative correlation. The study was well designed and enrolled both symptomatic and asymptomatic populations to provide sufficient statistical power for the comparative analysis of the methodologies, as well as to represent accurately the patient populations. This is a useful and timely study that has a great impact in clinical setting for the rapid detection of COVID-19.
-
Evaluation Summary:
The authors evaluate the performance of the Abbott ID NOW SARS-CoV-2 test in a group of non-hospitalized individuals being tested for COVID-19 and compared that performance to an RT-PCR test. The authors also provide an interesting review and meta-analysis of ID NOW performance across the literature. The cohort assessed in this study, however, was small and localized, which currently undermines its comparison with the known performance of the test in broader applications.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 agreed to share their name with the authors.)
-
-
SciScore for 10.1101/2020.12.07.20245225: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank…
SciScore for 10.1101/2020.12.07.20245225: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-