Effectiveness of rapid SARS-CoV-2 genome sequencing in supporting infection control for hospital-onset COVID-19 infection: Multicentre, prospective study

  1. Oliver Stirrup
  2. James Blackstone
  3. Fiona Mapp
  4. Alyson MacNeil
  5. Monica Panca
  6. Alison Holmes
  7. Nicholas Machin
  8. Gee Yen Shin
  9. Tabitha Mahungu
  10. Kordo Saeed
  11. Tranprit Saluja
  12. Yusri Taha
  13. Nikunj Mahida
  14. Cassie Pope
  15. Anu Chawla
  16. Maria-Teresa Cutino-Moguel
  17. Asif Tamuri
  18. Rachel Williams
  19. Alistair Darby
  20. David L Robertson
  21. Flavia Flaviani
  22. Eleni Nastouli
  23. Samuel Robson
  24. Darren Smith
  25. Matthew Loose
  26. Kenneth Laing
  27. Irene Monahan
  28. Beatrix Kele
  29. Sam Haldenby
  30. Ryan George
  31. Matthew Bashton
  32. Adam A Witney
  33. Matthew Byott
  34. Francesc Coll
  35. Michael Chapman
  36. Sharon J Peacock
  37. COG-UK HOCI Investigators
  38. The COVID-19 Genomics UK (COG-UK) consortium
  39. Joseph Hughes
  40. Gaia Nebbia
  41. David G Partridge
  42. Matthew Parker
  43. James Richard Price
  44. Christine Peters
  45. Sunando Roy
  46. Luke B Snell
  47. Thushan I de Silva
  48. Emma Thomson
  49. Paul Flowers
  50. Andrew Copas
  51. Judith Breuer

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This is a very extensive evaluation of the impact of rapid availability of whole-gemome sequencing results from SARS-CoV2 to inform infection control policies in hospital settings. It, most likely, is the largest analysis of its kind, clearly demonstrating the possibilities and challenges with this innovative technique.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #3 agreed to share their name with the authors.)

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Viral sequencing of SARS-CoV-2 has been used for outbreak investigation, but there is limited evidence supporting routine use for infection prevention and control (IPC) within hospital settings.

Methods:

We conducted a prospective non-randomised trial of sequencing at 14 acute UK hospital trusts. Sites each had a 4-week baseline data collection period, followed by intervention periods comprising 8 weeks of ‘rapid’ (<48 hr) and 4 weeks of ‘longer-turnaround’ (5–10 days) sequencing using a sequence reporting tool (SRT). Data were collected on all hospital-onset COVID-19 infections (HOCIs; detected ≥48 hr from admission). The impact of the sequencing intervention on IPC knowledge and actions, and on the incidence of probable/definite hospital-acquired infections (HAIs), was evaluated.

Results:

A total of 2170 HOCI cases were recorded from October 2020 to April 2021, corresponding to a period of extreme strain on the health service, with sequence reports returned for 650/1320 (49.2%) during intervention phases. We did not detect a statistically significant change in weekly incidence of HAIs in longer-turnaround (incidence rate ratio 1.60, 95% CI 0.85–3.01; p = 0.14) or rapid (0.85, 0.48–1.50; p = 0.54) intervention phases compared to baseline phase. However, IPC practice was changed in 7.8 and 7.4% of all HOCI cases in rapid and longer-turnaround phases, respectively, and 17.2 and 11.6% of cases where the report was returned. In a ‘per-protocol’ sensitivity analysis, there was an impact on IPC actions in 20.7% of HOCI cases when the SRT report was returned within 5 days. Capacity to respond effectively to insights from sequencing was breached in most sites by the volume of cases and limited resources.

Conclusions:

While we did not demonstrate a direct impact of sequencing on the incidence of nosocomial transmission, our results suggest that sequencing can inform IPC response to HOCIs, particularly when returned within 5 days.

Funding:

COG-UK is supported by funding from the Medical Research Council (MRC) part of UK Research & Innovation (UKRI), the National Institute of Health Research (NIHR) (grant code: MC_PC_19027), and Genome Research Limited, operating as the Wellcome Sanger Institute.

Clinical trial number:

NCT04405934 .

Article activity feed

  1. Version published to 10.7554/elife.78427 on eLife
  2. eLife

    Author Response

    Reviewer #1 (Public Review):

    The authors have provided an impressive analysis of the effects of reporting of WGS results on IPC practices in 14 hospitals in the UK during the COVID-19 pandemic. After a median of 4 weeks, hospitals adopted a practice of "rapid" or "longer" turnaround phases for WGS reporting. After a median of 8 weeks, 8 of 9 "rapid" hospitals adopted the "longer" practice for a median of 4 weeks. After a median of 4 weeks, all 5 "longer" hospitals adopted the "rapid" practice for a median 8 weeks. Hence, there were twice as many weeks with the rapid, compared to the longer reporting practice.

    The targeted turnaround times for reporting were 48 hours for the rapid and 5-10 days for the longer phase.

    The primary outcomes of the study were: (1) incidence of IPC-defined SARS-CoV-2 HAIs per week per 100 currently admitted non-COVID-19 inpatients, and (2) for each HOCI, identification of linkage to individuals within an outbreak of SARS-CoV-2 nosocomial transmission using sequencing data as interpreted through the SRT that was not identified by pre-sequencing IPC evaluation during intervention phases.

    Secondary outcomes were: (1) incidence of IPC-defined SARS-CoV-2 hospital outbreaks per week per 100 non-COVID-19 inpatients, (2) for each HOCI, any change to IPC actions following receipt of SRT report during intervention phases, (3) any recommended change to IPC actions (regardless of whether changes were implemented). The proportion of HOCI cases for which IPC reported the SRT report to be 'useful' was added as a further outcome.

    A total of 2170 HOCIs were recorded for the study between 15 October 2020 and 26 April 2021.

    The authors conclude that "While we did not demonstrate a direct impact of sequencing on the incidence of nosocomial transmission, our results suggest that sequencing can inform IPC response to HOCIs, particularly when returned within 5 days."

    The research question is very relevant and, as said, the amount of data collected is impressive. Yet, interpretation of the data, obtained in a real-life setting with all hurdles and complexities created by the pandemic situation, is challenging. I have several questions related to data interpretation and difficulties in accepting the overall positive interpretation of the findings when it comes to feasibility and potential impact. Especially, as I consider the real-time availability of WGS results in the participating hospitals to be (much) higher than it will be in hospitals in most other countries.

    We thank the reviewer for their summary and commentary on our work.

    Specific questions

    Not clear why sites started either with rapid or longer phase. Was there a randomization process? Please clarify.

    There was no randomisation process as the ordering of intervention phases was largely driven by logistical concerns that needed to be overcome in order to be able to run the intervention during a very unpredictable period of the pandemic. We have added the following text to the second paragraph of the Methods to clarify this:

    “The order of the intervention phases was pragmatically determined in some sites by the need to first run the ‘longer-turnaround’ phase to develop sample transport and sequencing procedures before attempting the ‘rapid’ sequencing phase, and the ordering was decided in the remaining sites to ensure a mixture of intervention phases over calendar time – there was no randomisation process in deciding the order of study phases.”

    From Figures S2 it is clear that the pandemic peaked, after which the curve declined when vaccination had started, and these curves seem to resemble the incidence rates of HOCI in the hospitals.

    Yes, there was also a full national lockdown in the UK in late 2020 and early 2021, which sharply reduced community incidence rates.

    I had difficulties in interpreting S3 and S4 where I think the authors incorporated these disease dynamics occurring outside the hospital setting on the HOCI incidences. I would be helped by a better explanation of what actually was done.

    Yes, this is correct. We have now added a more detailed model specification to the Appendix (also in response to the comments of Reviewer #3). We have also added some further explanation to the Figure captions: ‘The spline curves shown are estimated simultaneously within the final analysis model, and show how these factors have independent contributions to the prediction of the incidence rate for HAIs. The associations for each covariable indicated by model parameter point estimates are shown as solid lines, with 95%CIs shown as dashed lines.’

    It is not clear why the difference between the groups in the intervention (providing rapid or not so rapid WGS reports) was too small to have an impact, compared to the baseline period without WGS reporting. Surprisingly sites F and G appear to do significantly worse during the "rapid" phase, according to Fig1. Please clarify.

    With regards the issue of why it might be that we did not detect an effect of the intervention on the incidence rate of hospital-acquired infections, the Methods contains a relevant brief summary of our qualitative analysis within this study:

    “The SRT did provide new and valued insights into transmission events, outbreaks and wider hospital functioning but mainly acted to offer confirmation and reassurance to IPC teams. Critically, the capacity to generate and respond to these insights effectively on a case-by-case basis was breached in most sites by the volume of HOCIs, and the limits of finite human and physical resource (e.g. hospital layout). “

    We have now also added a sentence to flag this in the context of study limitations in the Discussion “Our qualitative analyses also found that the capacity of sites to react to information generated by the sequencing intervention was breached by the volume of HOCI and admitted COVID-19 cases in combination with the finite personnel resources and limited physical space for isolation that was available”.

    We have noted in the Results section that “Our analysis models reveal important findings beyond the effect of the intervention. The analysis model for the incidence of HAIs identified independent positive associations with the proportion of current SARS-CoV-2 positive inpatients, the local community incidence of new SARS-CoV-2 cases … and calendar time …”. The ‘rapid’ phases for sites F and G were conducted during periods of high community incidence of SARS-CoV-2, with a high proportion of current inpatients SARS-CoV-2 positive and at a relatively early stage in the national vaccination roll-out. These factors clearly outweighed any potential reduction in the incidence of HAIs associated with the sequencing intervention.

    We have added some further clarifications regarding the structure and interpretation of the analysis models for the outcome of the incidence of HAIs in response to comments from Reviewer #3.

    The 'health economic findings' miss the health component. The costs of the intervention are described in detail, but not the benefits of the intervention. Is it possible to calculate the costs required to prevent a single case of HOCI?

    Thank you for your comment. The scope of the economic evaluation was to evaluate the economic effects of SARS-CoV-2 genome sequencing in supporting infection control teams and not the health benefits of the intervention. Therefore, the outcome of the intervention would be rather the benefit of rapid/slow return of the sequencing report expressed as potential reduction in resource utilisation and costs. A paper presenting the methodology and findings is under preparation. Here we presented only the cost of the intervention. We will remove the “Health economic findings” heading and re-name it “Cost of SARS-CoV-2 genome sequencing”.

    Reviewer #2 (Public Review):

    This study evaluated the impact of rapid turnaround whole genome sequencing to discover unsuspected hospital-acquired SARS-CoV-2 on the incidence of hospital-acquired SARS-CoV-2 cases. Strengths of the study include the important question, the technical and logistic feat of making whole genome sequencing widely available, and the large number of participating sites.

    We thank the reviewer for their summary and commentary on our work.

    Major limitations of the study include the fact that only half of sequencing reports were returned to infection prevention programs and then only a small minority within the targeted reporting time-frame (5% for rapid phase, 21% for longer phase; median turnarounds were 5 days and 13 days respectively). This fundamentally undermines the premise of the study, namely to see if rapid turnaround of sequencing can impact infection control.

    We have now explicitly stated in the Results that the median turnaround times achieved were substantially longer than the target values, and have added that ‘…and more timely reporting of results might be associated with greater impact on IPC actions’ in the Discussion.

    More broadly, it does not appear that there was a standardized protocol on how hospitals were expected to respond to reports of clusters.

    Review of Table S2 suggests that many of the potential actions were things that in retrospect probably don't have too much impact on transmission (e.g. checking soap stocks, signage assessments). The kinds of things that I think might decrease nosocomial transmission include minimizing use of shared rooms, improving ventilation, increased use of N95/FFP2 respirators for source control, more frequent surveillance testing cadences, etc. These were not options on the response lists perhaps explaining the lack of impact on transmission.

    We have added the following paragraph to the Discussion: “Planning this study and developing the data collection forms during the early stages of a novel viral pandemic was challenging, as in the summer of 2020 there were still ongoing debates around the primary mode of viral transmission and optimal IPC practice, and global supply chains for personal protective equipment were strained. In the planning of an equivalent study now, there would be a greater focus on adjustments to ventilation, air filtration and respirator usage. It would also be possible to be more prescriptive and standardised regarding the recommended changes to IPC practice in response to sequencing findings.”

    Reviewer #3 (Public Review):

    This study, conducted in 14 acute hospital trusts in the United Kingdom, compared SARS-CoV-2 hospital infection outcomes in a four week baseline period with outcomes in periods with 'rapid' (<48h) and 'longer-turnaround' (5-10 day) sequencing with results fed-back to infection prevention and control teams using a bespoke sequencing reporting tool. The question of whether rapid sequencing of hospital-onset SARS-CoV-2 infection can, by informing infection prevention and control (IPC) actions, reduce nosocomial transmission is interesting and potentially important. To our knowledge, this study represents the first large-scale formal evaluation of such technology. While the results are, on the face of it, disappointing in that hospitals were largely unable to meet target turnaround times for sequencing and results provide no evidence of benefit of the intervention in reducing hospital-acquired infection (and in some cases, such as for the "hospital outbreaks" outcome, the confidence intervals are so wide as to be unable to rule out substantial benefits or harms of the intervention) the are a number of important strengths of the study. These include the relatively strong quasi-experimental design (a type of non-randomised cluster crossover), the pre-defined analysis plan, and adequate power for the primary outcomes.

    Limitations of the study include the practical difficulties that participating hospitals had in reporting sequencing results to the IPC teams in a timely manner that could be acted on and lack of sufficient consideration for the ways in which sequencing information could have directly informed IPC activities in ways that would have been likely to substantially reduce the spread of infection (for example, Table S2 reports changes to IPC as a result of sequencing reports which include generic activities such as "Assessment of alcogel stocks" or "IPC signage assessment", which seem like things which should be done anyway, and don't obviously depend on information from pathogen sequencing).

    We thank the reviewer for their summary and commentary on our work.

    There are also some aspects of transparency that need to be addressed: the analytical methods are not reported in sufficient detail to enable the work to be repeated, and the results are not reported with sufficient detail to an enable an assessment of the appropriateness or otherwise of the statistical models used in the analysis. Additionally, while the study protocol specified six secondary outcomes, not all of these are reported even where it appears that some (partial) information is available for unreported outcomes.

  3. eLife

    Evaluation Summary:

    This is a very extensive evaluation of the impact of rapid availability of whole-gemome sequencing results from SARS-CoV2 to inform infection control policies in hospital settings. It, most likely, is the largest analysis of its kind, clearly demonstrating the possibilities and challenges with this innovative technique.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #3 agreed to share their name with the authors.)

  4. eLife

    Reviewer #1 (Public Review):

    The authors have provided an impressive analysis of the effects of reporting of WGS results on IPC practices in 14 hospitals in the UK during the COVID-19 pandemic. After a median of 4 weeks, hospitals adopted a practice of "rapid" or "longer" turnaround phases for WGS reporting. After a median of 8 weeks, 8 of 9 "rapid" hospitals adopted the "longer" practice for a median of 4 weeks. After a median of 4 weeks, all 5 "longer" hospitals adopted the "rapid" practice for a median 8 weeks. Hence, there were twice as many weeks with the rapid, compared to the longer reporting practice.

    The targeted turnaround times for reporting were 48 hours for the rapid and 5-10 days for the longer phase.

    The primary outcomes of the study were: (1) incidence of IPC-defined SARS-CoV-2 HAIs per week per 100 currently admitted non-COVID-19 inpatients, and (2) for each HOCI, identification of linkage to individuals within an outbreak of SARS-CoV-2 nosocomial transmission using sequencing data as interpreted through the SRT that was not identified by pre-sequencing IPC evaluation during intervention phases.

    Secondary outcomes were: (1) incidence of IPC-defined SARS-CoV-2 hospital outbreaks per week per 100 non-COVID-19 inpatients, (2) for each HOCI, any change to IPC actions following receipt of SRT report during intervention phases, (3) any recommended change to IPC actions (regardless of whether changes were implemented). The proportion of HOCI cases for which IPC reported the SRT report to be 'useful' was added as a further outcome.

    A total of 2170 HOCIs were recorded for the study between 15 October 2020 and 26 April 2021.

    The authors conclude that "While we did not demonstrate a direct impact of sequencing on the incidence of nosocomial transmission, our results suggest that sequencing can inform IPC response to HOCIs, particularly when returned within 5 days."

    The research question is very relevant and, as said, the amount of data collected is impressive. Yet, interpretation of the data, obtained in a real-life setting with all hurdles and complexities created by the pandemic situation, is challenging. I have several questions related to data interpretation and difficulties in accepting the overall positive interpretation of the findings when it comes to feasibility and potential impact. Especially, as I consider the real-time availability of WGS results in the participating hospitals to be (much) higher than it will be in hospitals in most other countries.

    Specific questions
    Not clear why sites started either with rapid or longer phase. Was there a randomization process? Please clarify.

    From Figures S2 it is clear that the pandemic peaked, after which the curve declined when vaccination had started, and these curves seem to resemble the incidence rates of HOCI in the hospitals. I had difficulties in interpreting S3 and S4 where I think the authors incorporated these disease dynamics occurring outside the hospital setting on the HOCI incidences. I would be helped by a better explanation of what actually was done.

    It is not clear why the difference between the groups in the intervention (providing rapid or not so rapid WGS reports) was too small to have an impact, compared to the baseline period without WGS reporting. Surprisingly sites F and G appear to do significantly worse during the "rapid" phase, according to Fig1. Please clarify.

    The 'health economic findings' miss the health component. The costs of the intervention are described in detail, but not the benefits of the intervention. Is it possible to calculate the costs required to prevent a single case of HOCI?

  5. eLife

    Reviewer #2 (Public Review):

    This study evaluated the impact of rapid turnaround whole genome sequencing to discover unsuspected hospital-acquired SARS-CoV-2 on the incidence of hospital-acquired SARS-CoV-2 cases. Strengths of the study include the important question, the technical and logistic feat of making whole genome sequencing widely available, and the large number of participating sites. Major limitations of the study include the fact that only half of sequencing reports were returned to infection prevention programs and then only a small minority within the targeted reporting time-frame (5% for rapid phase, 21% for longer phase; median turnarounds were 5 days and 13 days respectively). This fundamentally undermines the premise of the study, namely to see if rapid turnaround of sequencing can impact infection control. More broadly, it does not appear that there was a standardized protocol on how hospitals were expected to respond to reports of clusters. Review of Table S2 suggests that many of the potential actions were things that in retrospect probably don't have too much impact on transmission (e.g. checking soap stocks, signage assessments). The kinds of things that I think might decrease nosocomial transmission include minimizing use of shared rooms, improving ventilation, increased use of N95/FFP2 respirators for source control, more frequent surveillance testing cadences, etc. These were not options on the response lists perhaps explaining the lack of impact on transmission.

  6. eLife

    Reviewer #3 (Public Review):

    This study, conducted in 14 acute hospital trusts in the United Kingdom, compared SARS-CoV-2 hospital infection outcomes in a four week baseline period with outcomes in periods with 'rapid' (<48h) and 'longer-turnaround' (5-10 day) sequencing with results fed-back to infection prevention and control teams using a bespoke sequencing reporting tool. The question of whether rapid sequencing of hospital-onset SARS-CoV-2 infection can, by informing infection prevention and control (IPC) actions, reduce nosocomial transmission is interesting and potentially important. To our knowledge, this study represents the first large-scale formal evaluation of such technology. While the results are, on the face of it, disappointing in that hospitals were largely unable to meet target turnaround times for sequencing and results provide no evidence of benefit of the intervention in reducing hospital-acquired infection (and in some cases, such as for the "hospital outbreaks" outcome, the confidence intervals are so wide as to be unable to rule out substantial benefits or harms of the intervention) the are a number of important strengths of the study. These include the relatively strong quasi-experimental design (a type of non-randomised cluster crossover), the pre-defined analysis plan, and adequate power for the primary outcomes.

    Limitations of the study include the practical difficulties that participating hospitals had in reporting sequencing results to the IPC teams in a timely manner that could be acted on and lack of sufficient consideration for the ways in which sequencing information could have directly informed IPC activities in ways that would have been likely to substantially reduce the spread of infection (for example, Table S2 reports changes to IPC as a result of sequencing reports which include generic activities such as "Assessment of alcogel stocks" or "IPC signage assessment", which seem like things which should be done anyway, and don't obviously depend on information from pathogen sequencing). There are also some aspects of transparency that need to be addressed: the analytical methods are not reported in sufficient detail to enable the work to be repeated, and the results are not reported with sufficient detail to an enable an assessment of the appropriateness or otherwise of the statistical models used in the analysis. Additionally, while the study protocol specified six secondary outcomes, not all of these are reported even where it appears that some (partial) information is available for unreported outcomes.

  7. ScreenIT

    SciScore for 10.1101/2022.02.10.22270799: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Ethicsnot detected.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Analyses were conducted using Stata V16, with figures generated using the ggplot2 package for R V4.0.
    ggplot2
    suggested: (ggplot2, RRID:SCR_014601)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    There are several limitations that may have impacted on the results from this study. The study was conducted between October 2020 and April 2021. In this period, the local community incidence for the study sites ranged from <50 to >1200 weekly cases per 100,000 people. There were corresponding large variations in the healthcare burden of COVID-19, with several sites recording weeks when more than half of all inpatients were SARS-CoV-2 positive. High community infection rates and associated increases in the incidence of HOCI cases contributed to difficulties for site research teams in generating good quality viral sequences and reports for all HOCI cases within target timeframes. It may therefore be more achievable to develop systems for rapid viral WGS and feedback for endemic respiratory viruses at lower and more consistent levels. The peak in SARS-CoV-2 levels in December 2020 to January 2021 corresponded to the rise of the highly transmissible Alpha variant in the UK[20]. We did not find that the local prevalence of the Alpha variant was associated with the incidence rate of HAIs, beyond any effect mediated by higher community incidence. This matches the conclusions of a previously reported sub-study analysis using data from our sites[28]. The study intervention made use of a bespoke sequence reporting tool[10]. The SRT combined both patient-meta-data and sequencing data, providing a single-page, easily interpretable report for IPC teams. It also facilitated standardisatio...

    Results from TrialIdentifier: We found the following clinical trial numbers in your paper:

    IdentifierStatusTitle
    NCT04405934CompletedCOG-UK Project Hospital-Onset COVID-19 Infections Study

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  8. Version published to 10.1101/2022.02.10.22270799v1 on medRxiv