Computer-Aided Medical Microbiology Monitoring Tool: A Strategy to Adapt to the SARS-CoV-2 Epidemic and That Highlights RT-PCR Consistency
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
Since the beginning of the COVID-19 pandemic, important health and regulatory decisions relied on SARS-CoV-2 reverse transcription polymerase chain reaction (RT-PCR) results. Our diagnostic laboratory faced a rapid increase in the number of SARS-CoV-2 RT-PCR. To maintain a rapid turnaround time, we moved from a case-by-case validation of RT-PCR results to an automated validation and immediate results transmission to clinicians. A quality-monitoring tool based on a homemade algorithm coded in R was developed, to preserve high quality and to track aberrant results. We present the results of this quality-monitoring tool applied to 35,137 RT-PCR results. Patients tested several times led to 4,939 pairwise comparisons: 88% concordant and 12% discrepant. The algorithm automatically solved 428 out of 573 discrepancies. The most likely explanation for these 573 discrepancies was related for 44.9% of the situations to the clinical evolution of the disease, 27.9% to preanalytical factors, and 25.3% to stochasticity of the assay. Finally, 11 discrepant results could not be explained, including 8 for which clinical data was not available. For patients repeatedly tested on the same day, the second result confirmed a first negative or positive result in 99.2% or 88.9% of cases, respectively. The implemented quality-monitoring strategy allowed to: i) assist the investigation of discrepant results ii) focus the attention of medical microbiologists onto results requiring a specific expertise and iii) maintain an acceptable turnaround time. This work highlights the high RT-PCR consistency for the detection of SARS-CoV-2 and the necessity for automated processes to handle a huge number of microbiological results while preserving quality.
Article activity feed
-
-
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume that …
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume that most of the results of patients tested only once met clinical expectations, supporting the adequacy of the provided analyses. Furthermore, results quality was insured by a technical validation. Such validation relies on: i) positive and negative controls integrated to each run, ii) the use of two RT-PCR targets and iii) the per-batch follow-up of the positivity rate; if a run would suddenly have a much higher rate of positivity would suggest a possible contamination problem with false positive. Additionally, the follow-up of the median quantity of virus was also used to identify a possible technical issue affecting the sensitivity. However, our work might be complemented by a systematic, unbiased repetition of analyses for quality purposes. Nevertheless, in the crisis we faced, the upfront repetition of analyses would have been impractical, and maybe even unethical, in a context of reagents and material shortage. Another limitation of our work is that while we intended to use an unbiased algorithm stable over time to investigate discrepancies in results, some of the applied criteria were partly arbitrary (e.g. the 10 days limit to consider discrepancies as expected due to the clinical evolution of the disease). Furthermore, our process consent the use o...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume …
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume that most of the results of patients tested only once met clinical expectations, supporting the adequacy of the provided analyses. Furthermore, results quality was insured by a technical validation. Such validation relies on: i) positive and negative controls integrated to each run, ii) the use of two RT-PCR targets and iii) the perbatch follow-up of the positivity rate; if a run would suddenly have a much higher rate of positivity would suggest a possible contamination problem with false positive. Additionally, the follow-up of the median quantity of virus was also used to identify a possible technical issue affecting the sensitivity. However, our work might be complemented by a systematic, unbiased repetition of analyses for quality purposes. Nevertheless, in the crisis we faced, the upfront repetition of analyses would have been impractical, and maybe even unethical, in a context of reagents and material shortage. Another limitation of our work is that while we intended to use an unbiased algorithm stable over time to investigate discrepancies in results, some of the applied criteria were partly arbitrary (e.g. the 10 days limit to consider discrepancies as expected due to the clinical evolution of the disease). Furthermore, our process consent the use of a single explanation for each observed discrepancy, while more could be applicable. While arguable, these choices were made to fit a strategy of quality monitoring. Indeed, the primary aim of the presented methodology was to attribute the observed discrepancies to the most likely explanation to focus on truly unexplainable and problematic cases. Clinical laboratory vulnerabilities during the COVID-19 pandemic were the subject of a recent publication by Lippi et al. (26). Our assessment overlaps with some of the preanalytical culprits identified by the authors such as specimen collection (see “detailed explanations” Tables S2 and S3). However, some other potential vulnerabilities were not considered as probable causes for discrepancies in our assessment, since they are covered by other pre-existing quality management procedures in our laboratory. For instance, our preanalytical team systematically rejects samples missing a patient identification. Moreover, in our laboratory, internal extraction controls and amplification controls are systematically included to detect samples that might contain interfering substances compromising the amplification (9). Repeated testing on identical samples is a recognized method for internal quality assurance, which can lead to the identification of systematic or sporadic vulnerabilities in microbiology laboratories (27). In the present quality monitoring, different samples from the same patient were compared, similarly to the “Delta check” routinely applied in clinical chemistry laboratories. However, microbiology laboratories present specific challenges due to the frequent occurrence of actual discrepant results drew by clinical evolution of the diseases, different sampling sites, robustness of the sampling procedure or stochasticity in presence of scarce DNA. These factors were considered by our monitoring tool to reduce the list of potentially problematic cases to 3 discrepant comparisons. Unfortunately, these 3 suspicious cases remained unaddressed at the time of this final analysis. If considered as erroneous results, 3 over the 35,349 analyses provided on the period of four months would be an acceptable error rate (0.01%), especially when compared to previous reports on error rates (0.1-9.3%) in medical laboratories data (28). Furthermore, similar cases would now, and thanks to our algorithm implemented in routine, lead to repetition of the analyses or feedback to the clinicians to obtain new samples. This is, to our knowledge, the first implementation in clinical microbiology of a quality monitor tool, similar to the “Delta check” used in clinical chemistry laboratories. We would recommend the implementation of a similar system for other laboratories facing the COVID-19 or other epidemic wave in order to help focusing on some specific results and to identify laboratory vulnerabilities. Furthermore, we hypothesized that similar longitudinal observation of results and algorithm-based selection of “cases to investigate” could also be applied to other highthroughput microbiology laboratory assays, either by the implementation of ad-hoc software as presented here or by rules embedded in the LIS. This work emphasized the benefit of an automatic algorithm capable of finding discrepant results and attributing them to corresponding testing phases. This computer-aided methodology outlined that besides the expected evolution of the disease, most of discrepant results are compatible with preanalytical factors. Moreover, most of URT samples collected repeatedly in a short timeframe showed consistent results, displaying the good reproducibility of the RT-PCR for SARS-CoV-2 detection. Application of this method for quality monitoring enabled to focus on problematic cases requiring biomedical expertise while maintaining an acceptable TAT.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
About SciScore
SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore is not a substitute for expert review. SciScore checks for the presence and correctness of RRIDs (research resource identifiers) in the manuscript, and detects sentences that appear to be missing RRIDs. SciScore also checks to make sure that rigor criteria are addressed by authors. It does this by detecting sentences that discuss criteria such as blinding or power analysis. SciScore does not guarantee that the rigor criteria that it detects are appropriate for the particular study. Instead it assists authors, editors, and reviewers by drawing attention to sections of the manuscript that contain or should contain various rigor criteria and key resources. For details on the results shown here, including references cited, please follow this link.
-
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume …
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume that most of the results of patients tested only once met clinical expectations, supporting the adequacy of the provided analyses. Furthermore, results quality was insured by a technical validation. Such validation relies on: i) positive and negative controls integrated to each run, ii) the use of two RT-PCR targets and iii) the perbatch follow-up of the positivity rate; if a run would suddenly have a much higher rate of positivity would suggest a possible contamination problem with false positive. Additionally, the follow-up of the median quantity of virus was also used to identify a possible technical issue affecting the sensitivity. However, our work might be complemented by a systematic, unbiased repetition of analyses for quality purposes. Nevertheless, in the crisis we faced, the upfront repetition of analyses would have been impractical, and maybe even unethical, in a context of reagents and material shortage. Another limitation of our work is that while we intended to use an unbiased algorithm stable over time to investigate discrepancies in results, some of the applied criteria were partly arbitrary (e.g. the 10 days limit to consider discrepancies as expected due to the clinical evolution of the disease). Furthermore, our process consent the use of a single explanation for each observed discrepancy, while more could be applicable. While arguable, these choices were made to fit a strategy of quality monitoring. Indeed, the primary aim of the presented methodology was to attribute the observed discrepancies to the most likely explanation to focus on truly unexplainable and problematic cases. Clinical laboratory vulnerabilities during the COVID-19 pandemic were the subject of a recent publication by Lippi et al. (26). Our assessment overlaps with some of the preanalytical culprits identified by the authors such as specimen collection (see “detailed explanations” Tables S2 and S3). However, some other potential vulnerabilities were not considered as probable causes for discrepancies in our assessment, since they are covered by other pre-existing quality management procedures in our laboratory. For instance, our preanalytical team systematically rejects samples missing a patient identification. Moreover, in our laboratory, internal extraction controls and amplification controls are systematically included to detect samples that might contain interfering substances compromising the amplification (9). Repeated testing on identical samples is a recognized method for internal quality assurance, which can lead to the identification of systematic or sporadic vulnerabilities in microbiology laboratories (27). In the present quality monitoring, different samples from the same patient were compared, similarly to the “Delta check” routinely applied in clinical chemistry laboratories. However, microbiology laboratories present specific challenges due to the frequent occurrence of actual discrepant results drew by clinical evolution of the diseases, different sampling sites, robustness of the sampling procedure or stochasticity in presence of scarce DNA. These factors were considered by our monitoring tool to reduce the list of potentially problematic cases to 3 discrepant comparisons. Unfortunately, these 3 suspicious cases remained unaddressed at the time of this final analysis. If considered as erroneous results, 3 over the 35,349 analyses provided on the period of four months would be an acceptable error rate (0.01%), especially when compared to previous reports on error rates (0.1-9.3%) in medical laboratories data (28). Furthermore, similar cases would now, and thanks to our algorithm implemented in routine, lead to repetition of the analyses or feedback to the clinicians to obtain new samples. This is, to our knowledge, the first implementation in clinical microbiology of a quality monitor tool, similar to the “Delta check” used in clinical chemistry laboratories. We would recommend the implementation of a similar system for other laboratories facing the COVID-19 or other epidemic wave in order to help focusing on some specific results and to identify laboratory vulnerabilities. Furthermore, we hypothesized that similar longitudinal observation of results and algorithm-based selection of “cases to investigate” could also be applied to other highthroughput microbiology laboratory assays, either by the implementation of ad-hoc software as presented here or by rules embedded in the LIS. This work emphasized the benefit of an automatic algorithm capable of finding discrepant results and attributing them to corresponding testing phases. This computer-aided methodology outlined that besides the expected evolution of the disease, most of discrepant results are compatible with preanalytical factors. Moreover, most of URT samples collected repeatedly in a short timeframe showed consistent results, displaying the good reproducibility of the RT-PCR for SARS-CoV-2 detection. Application of this method for quality monitoring enabled to focus on problematic cases requiring biomedical expertise while maintaining an acceptable TAT.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
About SciScore
SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore is not a substitute for expert review. SciScore checks for the presence and correctness of RRIDs (research resource identifiers) in the manuscript, and detects sentences that appear to be missing RRIDs. SciScore also checks to make sure that rigor criteria are addressed by authors. It does this by detecting sentences that discuss criteria such as blinding or power analysis. SciScore does not guarantee that the rigor criteria that it detects are appropriate for the particular study. Instead it assists authors, editors, and reviewers by drawing attention to sections of the manuscript that contain or should contain various rigor criteria and key resources. For details on the results shown here, including references cited, please follow this link.
-
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume …
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
A limitation of our quality surveillance methodology is that it leaves aside patients tested only once. However, repeated testing was recommended for results in disagreement with the clinical context. Thus, one can assume that most of the results of patients tested only once met clinical expectations, supporting the adequacy of the provided analyses. Furthermore, results quality was insured by a technical validation. Such validation relies on: i) positive and negative controls integrated to each run, ii) the use of two RT-PCR targets and iii) the perbatch follow-up of the positivity rate; if a run would suddenly have a much higher rate of positivity would suggest a possible contamination problem with false positive. Additionally, the follow-up of the median quantity of virus was also used to identify a possible technical issue affecting the sensitivity. However, our work might be complemented by a systematic, unbiased repetition of analyses for quality purposes. Nevertheless, in the crisis we faced, the upfront repetition of analyses would have been impractical, and maybe even unethical, in a context of reagents and material shortage. Another limitation of our work is that while we intended to use an unbiased algorithm stable over time to investigate discrepancies in results, some of the applied criteria were partly arbitrary (e.g. the 10 days limit to consider discrepancies as expected due to the clinical evolution of the disease). Furthermore, our process consent the use of a single explanation for each observed discrepancy, while more could be applicable. While arguable, these choices were made to fit a strategy of quality monitoring. Indeed, the primary aim of the presented methodology was to attribute the observed discrepancies to the most likely explanation to focus on truly unexplainable and problematic cases. Clinical laboratory vulnerabilities during the COVID-19 pandemic were the subject of a recent publication by Lippi et al. (26). Our assessment overlaps with some of the preanalytical culprits identified by the authors such as specimen collection (see “detailed explanations” Tables S2 and S3). However, some other potential vulnerabilities were not considered as probable causes for discrepancies in our assessment, since they are covered by other pre-existing quality management procedures in our laboratory. For instance, our preanalytical team systematically rejects samples missing a patient identification. Moreover, in our laboratory, internal extraction controls and amplification controls are systematically included to detect samples that might contain interfering substances compromising the amplification (9). Repeated testing on identical samples is a recognized method for internal quality assurance, which can lead to the identification of systematic or sporadic vulnerabilities in microbiology laboratories (27). In the present quality monitoring, different samples from the same patient were compared, similarly to the “Delta check” routinely applied in clinical chemistry laboratories. However, microbiology laboratories present specific challenges due to the frequent occurrence of actual discrepant results drew by clinical evolution of the diseases, different sampling sites, robustness of the sampling procedure or stochasticity in presence of scarce DNA. These factors were considered by our monitoring tool to reduce the list of potentially problematic cases to 3 discrepant comparisons. Unfortunately, these 3 suspicious cases remained unaddressed at the time of this final analysis. If considered as erroneous results, 3 over the 35,349 analyses provided on the period of four months would be an acceptable error rate (0.01%), especially when compared to previous reports on error rates (0.1-9.3%) in medical laboratories data (28). Furthermore, similar cases would now, and thanks to our algorithm implemented in routine, lead to repetition of the analyses or feedback to the clinicians to obtain new samples. This is, to our knowledge, the first implementation in clinical microbiology of a quality monitor tool, similar to the “Delta check” used in clinical chemistry laboratories. We would recommend the implementation of a similar system for other laboratories facing the COVID-19 or other epidemic wave in order to help focusing on some specific results and to identify laboratory vulnerabilities. Furthermore, we hypothesized that similar longitudinal observation of results and algorithm-based selection of “cases to investigate” could also be applied to other highthroughput microbiology laboratory assays, either by the implementation of ad-hoc software as presented here or by rules embedded in the LIS. This work emphasized the benefit of an automatic algorithm capable of finding discrepant results and attributing them to corresponding testing phases. This computer-aided methodology outlined that besides the expected evolution of the disease, most of discrepant results are compatible with preanalytical factors. Moreover, most of URT samples collected repeatedly in a short timeframe showed consistent results, displaying the good reproducibility of the RT-PCR for SARS-CoV-2 detection. Application of this method for quality monitoring enabled to focus on problematic cases requiring biomedical expertise while maintaining an acceptable TAT.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
About SciScore
SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore is not a substitute for expert review. SciScore checks for the presence and correctness of RRIDs (research resource identifiers) in the manuscript, and detects sentences that appear to be missing RRIDs. SciScore also checks to make sure that rigor criteria are addressed by authors. It does this by detecting sentences that discuss criteria such as blinding or power analysis. SciScore does not guarantee that the rigor criteria that it detects are appropriate for the particular study. Instead it assists authors, editors, and reviewers by drawing attention to sections of the manuscript that contain or should contain various rigor criteria and key resources. For details on the results shown here, including references cited, please follow this link.
-
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement Furthermore, our process consent the use of a single explanation for each observed discrepancy, while more could be applicable. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
Data from additional tools added to each annotation on a weekly basis.
About SciScore
SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore is not a substitute for expert review. SciScore checks for the presence and correctness of RRIDs …
SciScore for 10.1101/2020.07.27.20162123: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement Furthermore, our process consent the use of a single explanation for each observed discrepancy, while more could be applicable. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
Data from additional tools added to each annotation on a weekly basis.
About SciScore
SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore is not a substitute for expert review. SciScore checks for the presence and correctness of RRIDs (research resource identifiers) in the manuscript, and detects sentences that appear to be missing RRIDs. SciScore also checks to make sure that rigor criteria are addressed by authors. It does this by detecting sentences that discuss criteria such as blinding or power analysis. SciScore does not guarantee that the rigor criteria that it detects are appropriate for the particular study. Instead it assists authors, editors, and reviewers by drawing attention to sections of the manuscript that contain or should contain various rigor criteria and key resources. For details on the results shown here, including references cited, please follow this link.
-
