Reliability of multi-site UK Biobank MRI brain phenotypes for the assessment of neuropsychiatric complications of SARS-CoV-2 infection: The COVID-CNS travelling heads study

  1. Eugene Duff
  2. Fernando Zelaya
  3. Fidel Alfaro Almagro
  4. Karla L. Miller
  5. Naomi Martin
  6. Thomas E. Nichols
  7. Bernd Taschler
  8. Ludovica Griffanti
  9. Christoph Arthofer
  10. Gwenaëlle Douaud
  11. Chaoyue Wang
  12. Thomas W. Okell
  13. Richard A. I. Bethlehem
  14. Klaus Eickel
  15. Matthias Günther
  16. David K. Menon
  17. Guy Williams
  18. Bethany Facer
  19. David J. Lythgoe
  20. Flavio Dell’Acqua
  21. Greta K. Wood
  22. Steven C. R. Williams
  23. Gavin Houston
  24. Simon S. Keller
  25. Catherine Holden
  26. Monika Hartmann
  27. Lily George
  28. Gerome Breen
  29. Benedict D. Michael
  30. Peter Jezzard
  31. Stephen M. Smith
  32. Edward T. Bullmore
  33. on behalf of the COVID-CNS Consortium

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Magnetic resonance imaging (MRI) of the brain could be a key diagnostic and research tool for understanding the neuropsychiatric complications of COVID-19. For maximum impact, multi-modal MRI protocols will be needed to measure the effects of SARS-CoV-2 infection on the brain by diverse potentially pathogenic mechanisms, and with high reliability across multiple sites and scanner manufacturers. Here we describe the development of such a protocol, based upon the UK Biobank, and its validation with a travelling heads study. A multi-modal brain MRI protocol comprising sequences for T1-weighted MRI, T2-FLAIR, diffusion MRI (dMRI), resting-state functional MRI (fMRI), susceptibility-weighted imaging (swMRI), and arterial spin labelling (ASL), was defined in close approximation to prior UK Biobank (UKB) and C-MORE protocols for Siemens 3T systems. We iteratively defined a comparable set of sequences for General Electric (GE) 3T systems. To assess multi-site feasibility and between-site variability of this protocol, N = 8 healthy participants were each scanned at 4 UK sites: 3 using Siemens PRISMA scanners (Cambridge, Liverpool, Oxford) and 1 using a GE scanner (King’s College London). Over 2,000 Imaging Derived Phenotypes (IDPs), measuring both data quality and regional image properties of interest, were automatically estimated by customised UKB image processing pipelines (S2 File). Components of variance and intra-class correlations (ICCs) were estimated for each IDP by linear mixed effects models and benchmarked by comparison to repeated measurements of the same IDPs from UKB participants. Intra-class correlations for many IDPs indicated good-to-excellent between-site reliability. Considering only data from the Siemens sites, between-site reliability generally matched the high levels of test-retest reliability of the same IDPs estimated in repeated, within-site, within-subject scans from UK Biobank. Inclusion of the GE site resulted in good-to-excellent reliability for many IDPs, although there were significant between-site differences in mean and scaling, and reduced ICCs, for some classes of IDP, especially T1 contrast and some dMRI-derived measures. We also identified high reliability of quantitative susceptibility mapping (QSM) IDPs derived from swMRI images, multi-network ICA-based IDPs from resting-state fMRI, and olfactory bulb structure IDPs from T1, T2-FLAIR and dMRI data.

Conclusion

These results give confidence that large, multi-site MRI datasets can be collected reliably at different sites across the diverse range of MRI modalities and IDPs that could be mechanistically informative in COVID brain research. We discuss limitations of the study and strategies for further harmonisation of data collected from sites using scanners supplied by different manufacturers. These acquisition and analysis protocols are now in use for MRI assessments of post-COVID patients (N = 700) as part of the ongoing COVID-CNS study.

Article activity feed

  1. Version published to 10.1371/journal.pone.0273704
  2. ScreenIT

    SciScore for 10.1101/2021.10.13.21264967: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Image processing pipelines and IDPs: Each MRI modality was analysed using custom pipelines for image pre-processing and estimation of multiple MRI contrast metrics and imaging-derived phenotypes (IDPs) derived from the UKB analysis pipelines (www.fmrib.ox.ac.uk/ukbiobank/) (Alfaro-Almagro et al., 2018) and software tools from the FMRIB Software Library (Jenkinson et al., 2012).
    FMRIB
    suggested: (FSL, RRID:SCR_002823)
    T1-weighted and T2-FLAIR images were combined in FreeSurfer to model the cortical surface (Desikan et al., 2006; Fischl et al., 2004).
    FreeSurfer
    suggested: (FreeSurfer, RRID:SCR_001847)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    We considered that the generally good-to-very good levels of reliability for most IDPs across all sites in this pilot study were sufficient to support this more inclusive strategy of using sites with either Siemens or GE scanners, with the caveat that this will entail loss of power to detect case-control differences in terms of IDPs derived from ASL and other modalities which were most difficult to harmonize between manufacturers. Between-site offsets in the mean and scaling of IDP values could be corrected statistically post hoc by standard harmonisation or modelling methods such as COMBAT(Da-ano et al., 2020) or Generalised Additive Modelling (Dinga et al., 2021), so long as certain sampling requirements for patients and controls can be achieved at individual sites. Methodological issues: It is a strength of this study that we have assessed reliability across a wide range of MRI modalities and imaging-derived phenotypes, using data collected from different MRI systems and at different sites. It is also a strength that we have been able to benchmark between-site reliability for the majority of IDPs against comparable estimates of test-retest reliability in the UKB data. However, sample size for the travelling heads study was small, meaning that results were potentially vulnerable to the effects of 1 or 2 outlying observations and confidence intervals were generally wide. We made best efforts, under the pragmatic constraints of urgently responding to a public health crisis, t...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.10.13.21264967v1 on medRxiv