Toward a more informative representation of the fetal–neonatal brain connectome using variational autoencoder

  1. Jung-Hoon Kim
  2. Josepheen De Asis-Cruz
  3. Dhineshvikram Krishnamurthy
  4. Catherine Limperopoulos

  • Curated by eLife

    eLife logo

    eLife assessment

    This study presents an application of a deep learning approach (adult-trained variational autoencoder) to describe the development of the functional brain connectome in human fetuses and neonates. The results suggest that this may lead to a better characterization of the complex patterns of brain maturation during this period. The evidence is convincing but the impact of other confounding factors in addition to maturation on the results could be explored and further analysis should be considered to highlight how this method can account for non-linear patterns of development, as well as the biological plausibility of the observed brain states. This work is of potential methodological interest to researchers exploring functional brain networks and brain development notably with deep learning.

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Recent advances in functional magnetic resonance imaging (fMRI) have helped elucidate previously inaccessible trajectories of early-life prenatal and neonatal brain development. To date, the interpretation of fetal–neonatal fMRI data has relied on linear analytic models, akin to adult neuroimaging data. However, unlike the adult brain, the fetal and newborn brain develops extraordinarily rapidly, far outpacing any other brain development period across the life span. Consequently, conventional linear computational models may not adequately capture these accelerated and complex neurodevelopmental trajectories during this critical period of brain development along the prenatal-neonatal continuum. To obtain a nuanced understanding of fetal–neonatal brain development, including nonlinear growth, for the first time, we developed quantitative, systems-wide representations of brain activity in a large sample (>500) of fetuses, preterm, and full-term neonates using an unsupervised deep generative model called variational autoencoder (VAE), a model previously shown to be superior to linear models in representing complex resting-state data in healthy adults. Here, we demonstrated that nonlinear brain features, that is, latent variables, derived with the VAE pretrained on rsfMRI of human adults, carried important individual neural signatures, leading to improved representation of prenatal-neonatal brain maturational patterns and more accurate and stable age prediction in the neonate cohort compared to linear models. Using the VAE decoder, we also revealed distinct functional brain networks spanning the sensory and default mode networks. Using the VAE, we are able to reliably capture and quantify complex, nonlinear fetal–neonatal functional neural connectivity. This will lay the critical foundation for detailed mapping of healthy and aberrant functional brain signatures that have their origins in fetal life.

Article activity feed

  1. Version published to 10.7554/elife.80878 on eLife
  2. eLife

    Author Response

    Reviewer #3 (Public Review):

    In this manuscript, Kim et al. use a deep generative model (a Variational Auto Encoder previously applied to adult data) to characterize neonatal-fetal functional brain development. The authors suggest that this approach is suitable given the rapid non-linear development taking place in the human brain across this period. Using two large neonatal and one fetal datasets, they describe that the resultant latent variables can lead to improved characterization of prenatal-neonatal development patterns, stable age prediction and that the decoder can reveal resting state networks. The study uses already accessible public datasets and the methods have been also made available.

    The manuscript is clearly written, the figures excellent and the application in this group novel. The methods are generally appropriate although there are some methodological concerns which I think would be important to address. Although the authors demonstrate that the methods are broadly generalisable across study populations - however, I am unsure about the general interest of the work beyond application of their previously described VAE approach to a new population and what new insight this offers to understanding how the human brain develops. This is a particular consideration given that the major results are age prediction (which is easily done with various imaging measures including something as simple as whole brain volume) and recapitulation of known patterns of functional activity in neonates. As such, the work will be of interest to researchers working in fMRI analysis methods and deep learning, but perhaps less so to a wider neuroscience/clinical readership.

    Specific comments:

    1. (M1) If I understand correctly, the method takes the functional data after volume registration into template space and then projects this data onto the surface. Given the complexities of changing morphology of the development brain. would it not be preferable to have the data in surface space for standard space alignment (rather than this being done later?). This would certainly help with one of the concerns expressed by the authors of "smoothing" in the youngest fetuses leading to a negative relationship between age and performance.

    While projecting onto the cortical surface has its advantages, as suggested here18, several studies have also shown that with careful registration, such as in the current study, volumetric registration can yield comparable performance19. Regardless, we did attempt to directly generate cortical surfaces for our fetuses. We refer the reviewer to our response to the RE-M2 [page 9].

    Regarding the “smoothing” effect in the youngest fetuses, we want to clarify that the smoothing effect in the scans of young fetuses is not unique to the choice of registration method. In other words, the same smoothing effect must be seen with cortical registration as well. Regarding this perspective, we kindly refer the reviewer to our response to RE-M1 [page 7]. Regarding the specific change made in the revised manuscript, we kindly refer to our response to R1-m5 [p21] or [page 9 line 191-213] in the main manuscript.

    1. (M2) A key limitation which I feel is important to consider if the method is aiming to be used for fetuses is the effects of the analysis being limited only to the cortical surface - and therefore the role of subcortical tissue (such as developmental layers in the immature white matter and key structures like the thalami) cannot be included. This is important, as in the fetal (and preterm neonatal) brain, the cortex is still developing and so not only might there be not the same kind of organisation to the activity, but also there is likely an evolving relationship with activity in the transient developmental layers (like the subplate) and inputs from the thalamus.

    The reviewer raises an important point. We agree with the reviewer that the subcortical region plays a critical role in fetal and newborn neurodevelopment. Unfortunately, our current VAE model cannot utilize such information without a major change in the model structure. We added this as a limitation of our study and discussed why our VAE model, in its current form, did not include subcortical areas. Please see our detailed response to RE-M1 [page 4] or [page 25 line 558-570] in the main manuscript.

    1. (M3) As the authors correctly describe, brain development and specifically functional relationships are likely evolving across the study time window. Beyond predicting age and a different way of estimating resting state networks using the decoding step, it is not clear to me what new insight the work is adding to the existing literature - or how the method has been specifically adapted for working with this kind of data. Whilst I agree that these developmental processes are indeed likely non-linear, to put the work in context, I think the manuscript would benefit from explaining how (or if) the method has been adapted and explicitly mentioning what additional neuroscientific/biological gains there are from this method.

    We appreciate the reviewer’s critical insights. In the revised paper, we included additional results that, we hope, can address the reviewer’s concerns. We believe that the strength of the VAE model is that, relative to linear models, it can be more generalizable across different datasets and ages (adult vs. full-term babies vs. preterm babies vs. fetuses). In the original manuscript, this was supported by the superior age prediction performance of the VAE over linear models when applied to different datasets covering the fetal to neonatal periods. Age prediction could also be done using other imaging modalities, as the reviewer pointed out. However, we do not think this undermines the potential impact of having the ability to accurately estimate age based on functional connectivity patterns. Brain function-structure relationships may not exactly be one-to-one20. It is entirely possible that for one disease, brain functional connectivity alterations precede structural changes such that delayed growth trajectories will first manifest in the functional space. There are also certain aspects of brain function that cannot be mapped directly to its structural characteristics (i.e., structural connectivity patterns). For example, brain changes its functional connectivity patterns dynamically over different brain states (resting vs. task-engaging)21, mental disorders (depression22, anxiety23, Schizophrenia24), cognitive traits25, 26, and individual uniqueness25, etc. Therefore, we believe that estimating the functional age of fetuses and neonates given their functional connectivity profiles may provide a biomarker for tracking neurodevelopment trajectories, allowing clinicians to identify deviations early and intervene in a timely manner if necessary. For these reasons, we believe that superior age prediction performance of the VAE model compared to linear models is scientifically significant.

    The value of the VAE lies in its ability to capture FC features that are otherwise not modeled by linear strategies. For example, here, we showed that only the VAE model can extract latent variables representing brain networks that are similar across different datasets. In contrast, linear models, showed higher network pattern similarity between full-term and preterm infants within the dHCP dataset. This suggests that the VAE model can be a very useful tool for capturing common brain networks in datasets acquired using different recording parameters and preprocessing steps. Moreover, the VAE representations predicted age with higher accuracy compared to linear representations. Together, these findings show that the methodology is effective in extracting functionally relevant features of the brain. Please see RE-M1 [page 3] and R1-m13 regarding the specific changes made in the revised manuscript.

    1. (M4) The unavoidable smoothing effect of VAE is very noticeable in the figures - does this suggest that the method will be relatively insensitive to the fine granularity which is important to understand brain development and the establishment of networks (such as the evolving boundaries between functional regions with age) - reducing inference to only the large primary sensory and associative networks? This will also be important to consider for the individual "reconstruction degree" - (which it would likely then overstate - and would need careful intersubject comparison also) if it was to be used as a biomarker or predictor of cognition as suggested by the authors.

    Regarding the first concern, yes. Greater smoothing will tend to yield less granular network patterns; this is true for all representational models (not only VAE, but also models like ICA or PCA). This effect becomes ever more pronounced when representations consist of fewer components (e.g., IC50); the smoothing effect becomes stronger, leading to coarser brain patterns (see Fig. 3 in the revised manuscript). In this regard, higher number of components is desired, but on the flipside, IC maps with higher components are generally less interpretable. In short, there will always be trade-offs between interpretability and spatial resolution. Also, higher components tend to cause over-fitting issue, as shown in our age prediction performance across different datasets (worse performance in the IC300 vs. IC50). In this sense, what matters for the representations is how informative each latent variable (or component) is. In the revised Fig. 2, we showed that latent variables from the VAE model were more informative in representing rsfMRI than linear representations. It is also noteworthy that the smoothing effect of the VAE is comparable to IC300 (similar effect to manual smoothing at the level of FWHM=5mm; revised Fig. 3). Given above results, we believe the VAE model may be more suitable for investigating finer scale of brain networks, than linear models. The above perspective was updated in the revised manuscript as [page 23 line 506-511]:

    "Another interesting observation was that the smoothing effect of the VAE is comparable to IC300 (similar effect to manual smoothing at the level of FWHM=5mm; Fig. 3). Given the above, we believe the VAE model may be more suitable for investigating finer scale of brain networks, than linear models. Perhaps, the VAE model with a greater number of latent variables (e.g., 512 or 1024 instead of 256 in the current VAE) can be utilized to find brain networks at finer scale."

    On top of the points raised above, network mapping with linear models is limited when it comes to mapping the spatial evolution of brain networks over aging due to their linear nature. This limitation can be observed in the ICA study with dHCP dataset (Fig. 4 in 7). On the other hand, thanks to its nonlinearity nature, the VAE model may have a potential to observe the spatial gradient of brain network over aging, while this expectation needs confirmation. To that end, we revised our discussion to reflect our perspective. We refer the full change made in the revised manuscript to our response to R1-m13.

  3. eLife

    eLife assessment

    This study presents an application of a deep learning approach (adult-trained variational autoencoder) to describe the development of the functional brain connectome in human fetuses and neonates. The results suggest that this may lead to a better characterization of the complex patterns of brain maturation during this period. The evidence is convincing but the impact of other confounding factors in addition to maturation on the results could be explored and further analysis should be considered to highlight how this method can account for non-linear patterns of development, as well as the biological plausibility of the observed brain states. This work is of potential methodological interest to researchers exploring functional brain networks and brain development notably with deep learning.

  4. eLife

    Reviewer #1 (Public Review):

    In this manuscript, the authors employed an adult-trained variational autoencoder deep learning model on a relatively large sample (over 700) of human fetal-neonatal resting fMRI data to enhance the individual non-linear compression of functional activity patterns of baby brains. This approach showed better performance in the reconstruction of functional fluctuation maps, age prediction accuracy, and age prediction generalizability in fetal and neonatal fMRI data compared with conventional linear models such as spatial independent component analysis. This method also revealed distinct baby brain functional networks spanning primary and high-level systems.

    This is an inspired attempt to represent non-linear changes in fetal-neonatal brain fMRI data. Considering the high noises and inconsistent functional spatial distributions in baby fMRI images, stable and sensitive feature extraction approaches are urgently needed in the field of early brain studies. This work is well designed and well written in general.

  5. eLife

    Reviewer #2 (Public Review):

    In the paper, the authors aimed to repurpose a previously developed Variational Autoencoder (VAE) trained on adult rsfMRI data to characterise the in vivo foetal-neonatal brain development. Although the attempts to understand both healthy and aberrant early functional development are becoming increasingly popular, the processing and interpretation of the foetal-neonatal rsfMRI remain challenging due to methodological difficulties and the extremely fast and complex nature of the early brain development itself. For this reason, the non-linear computational models, such as the proposed VAE, have the potential to represent the rsfMRI data and capture the early neurodevelopmental trajectories with higher accuracy compared to more prevalent linear methods such as ICA.

    In this vein, the authors successfully apply the adult-trained VAE to compress the spatial representation of foetal-neonate rsfMRI cortical patterns into 256 latent features. Due to the non-linear nature of the VAE, this latent representation has the potential to yield more informative brain representations of rsfMRI data compared to other available methods making it a strength of the article.

    Nevertheless, one important limitation is that the direct application of the model trained on adult data to early functional connectome and more importantly, the interpretation of the reconstructed latent space-based maps rests on a strong assumption that the adult connectome features are stable and recognisable in the very early period. Moreover, such a model trained on the adult data would also be incapable to reveal possible network structures that would be present in the developing but not in the adult brain.

    The attempt to validate the method and assess its generalisability on two independent, fairly large datasets that include foetuses, and preterm- and term-born infants is commendable. However, the interpretation of the results in light of the subject, image acquisition, and processing (which is widely recognised to be very difficult, especially in foetuses) heterogeneity requires caution. For example, the VAE reconstruction error is positively correlated with the age at scan in dHCP, and DBI full-terms, but the relationship is very strong in the reverse direction in DBI foetuses. This suggests differences between the subgroups of subjects which might be driven by factors other than age. Thus, we cannot exclude the possibility that the high age-predictive power of the models based on the latent features is partly driven by those differences in addition to the age-dependant features of the infant functional connectome.

    The approach for the extraction and mapping of the group-level brain resting state networks is interesting and has the potential to uncover new insights into the early connectome. However, some of the current results are rather surprising and put into question their biological plausibility. For example, the authors suggest observing the precursor of the default-mode network in the DBI but not the dHCP dataset. This is rather strange given the DBI subjects (including foetuses) were on average scanned earlier than the dHCP subjects. Also, the pattern similarity of the best matched extracted independent component ('brain network') in the full-term dHCP vs full-term DBI comparison is 0.6 which is rather low if expecting the same networks to be extracted in the age-matched comparison. Additionally, the network visualisations show large heterogeneity of the distribution of activation/deactivations within extracted independent components between the datasets (even after ordering them for pattern similarity) which contradicts the expectation that the extracted networks (if real) should be stable, if not along the whole development, then at least between the narrower age ranges within the datasets.

    Overall, the interpretation of the current work is somewhat limited, and careful analysis of the latent representations derived from foetal-neonate data might be required to dissociate the effects of potential confounders from biological/developmental mechanisms. This might be difficult in the context of the highly complex and mostly black-box strategy such as VAE (this applies not only to the current method but to all novel methods proposed to study rsfMRI). Despite these limitations, the proposed approach could be very interesting methodologically with a potential impact on the future analysis of rsfMRI data. Overall, the authors achieved their aim of applying a novel VAE method to foetal-neonatal functional data and demonstrated that the extracted latent variables are predictive of brain age. However, careful evaluation of the latent representations and differences in predictive results and the mapped networks between the two datasets might be necessary to support the conclusion that the VAE-derived representations of foetal-neonatal rsfMRI carry informative neural signatures.

  6. eLife

    Reviewer #3 (Public Review):

    In this manuscript, Kim et al. use a deep generative model (a Variational Auto Encoder previously applied to adult data) to characterize neonatal-fetal functional brain development. The authors suggest that this approach is suitable given the rapid non-linear development taking place in the human brain across this period. Using two large neonatal and one fetal datasets, they describe that the resultant latent variables can lead to improved characterization of prenatal-neonatal development patterns, stable age prediction and that the decoder can reveal resting state networks. The study uses already accessible public datasets and the methods have been also made available.

    The manuscript is clearly written, the figures excellent and the application in this group novel. The methods are generally appropriate although there are some methodological concerns which I think would be important to address. Although the authors demonstrate that the methods are broadly generalisable across study populations - however, I am unsure about the general interest of the work beyond application of their previously described VAE approach to a new population and what new insight this offers to understanding how the human brain develops. This is a particular consideration given that the major results are age prediction (which is easily done with various imaging measures including something as simple as whole brain volume) and recapitulation of known patterns of functional activity in neonates. As such, the work will be of interest to researchers working in fMRI analysis methods and deep learning, but perhaps less so to a wider neuroscience/clinical readership.

    Specific comments:
    1. If I understand correctly, the method takes the functional data after volume registration into template space and then projects this data onto the surface. Given the complexities of changing morphology of the development brain. would it not be preferable to have the data in surface space for standard space alignment (rather than this being done later?). This would certainly help with one of the concerns expressed by the authors of "smoothing" in the youngest fetuses leading to a negative relationship between age and performance.
    2. A key limitation which I feel is important to consider if the method is aiming to be used for fetuses is the effects of the analysis being limited only to the cortical surface - and therefore the role of subcortical tissue (such as developmental layers in the immature white matter and key structures like the thalami) cannot be included. This is important, as in the fetal (and preterm neonatal) brain, the cortex is still developing and so not only might there be not the same kind of organisation to the activity, but also there is likely an evolving relationship with activity in the transient developmental layers (like the subplate) and inputs from the thalamus.
    3. As the authors correctly describe, brain development and specifically functional relationships are likely evolving across the study time window. Beyond predicting age and a different way of estimating resting state networks using the decoding step, it is not clear to me what new insight the work is adding to the existing literature - or how the method has been specifically adapted for working with this kind of data. Whilst I agree that these developmental processes are indeed likely non-linear, to put the work in context, I think the manuscript would benefit from explaining how (or if) the method has been adapted and explicitly mentioning what additional neuroscientific/biological gains there are from this method.
    4. The unavoidable smoothing effect of VAE is very noticeable in the figures - does this suggest that the method will be relatively insensitive to the fine granularity which is important to understand brain development and the establishment of networks (such as the evolving boundaries between functional regions with age) - reducing inference to only the large primary sensory and associative networks? This will also be important to consider for the individual "reconstruction degree" - (which it would likely then overstate - and would need careful intersubject comparison also) if it was to be used as a biomarker or predictor of cognition as suggested by the authors.

  7. Version published to 10.1101/2022.06.16.496454v1 on bioRxiv