Evidence for embracing normative modeling
Curation statements for this article:-
Curated by eLife
eLife assessment
This is a rigorous and compelling extension of previous normative modeling work that demonstrates that normative models incorporating lifespan trajectories of structural and functional connectivity provide a strong basis for brain imaging studies across a range of tasks including, univariate group difference assessment, classification, and building regression models. The work is important, rigorous and a valuable contribution to the field.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Preprints for generating images with AI (BlueReZZ)
Abstract
In this work, we expand the normative model repository introduced in Rutherford et al., 2022a to include normative models charting lifespan trajectories of structural surface area and brain functional connectivity, measured using two unique resting-state network atlases (Yeo-17 and Smith-10), and an updated online platform for transferring these models to new data sources. We showcase the value of these models with a head-to-head comparison between the features output by normative modeling and raw data features in several benchmarking tasks: mass univariate group difference testing (schizophrenia versus control), classification (schizophrenia versus control), and regression (predicting general cognitive ability). Across all benchmarks, we show the advantage of using normative modeling features, with the strongest statistically significant results demonstrated in the group difference testing and classification tasks. We intend for these accessible resources to facilitate the wider adoption of normative modeling across the neuroimaging community.
Article activity feed
-
-
eLife assessment
This is a rigorous and compelling extension of previous normative modeling work that demonstrates that normative models incorporating lifespan trajectories of structural and functional connectivity provide a strong basis for brain imaging studies across a range of tasks including, univariate group difference assessment, classification, and building regression models. The work is important, rigorous and a valuable contribution to the field.
-
Reviewer #1 (Public Review):
The authors present normative modeling results using both structural data and functional connectivity data to demonstrate the strength of normative modeling in investigations of group effects, classification tasks, and brain-behavioral modeling. The models are built across 3 large data sets and tested in a rigorous manner. The strengths of this work are in the clarity or presentation, the demonstration of the value of normative modeling, the availability of the models and code, and the statistical rigor supporting the results. The work will have a significant impact on the field in that such models (built in large data sets) can be applied to smaller studies of specific populations of interest, therefore, facilitating research on many populations in a statistically rigorous manner.
-
Reviewer #2 (Public Review):
This work provides a direct extension of the authors' previously published paper "Charting brain growth and aging at high spatial precision" (Rutherford et al. 2022), expanding their highly valuable existing repository of pre-trained normative models to now also include cortical thickness, surface area, and functional connectivity data.
Strengths
Building on previously published and validated methodology, this work significantly expands an existing modelling toolbox with new data modalities, particularly functional connectivity measures.Model comparisons show that deviation scores derived from normative models perform as well, or better than, raw data models across three different benchmarking tests (group differences, classification, regression). The authors clearly demonstrate the utility of deviation …
Reviewer #2 (Public Review):
This work provides a direct extension of the authors' previously published paper "Charting brain growth and aging at high spatial precision" (Rutherford et al. 2022), expanding their highly valuable existing repository of pre-trained normative models to now also include cortical thickness, surface area, and functional connectivity data.
Strengths
Building on previously published and validated methodology, this work significantly expands an existing modelling toolbox with new data modalities, particularly functional connectivity measures.Model comparisons show that deviation scores derived from normative models perform as well, or better than, raw data models across three different benchmarking tests (group differences, classification, regression). The authors clearly demonstrate the utility of deviation scores in the assessment of both group and individual differences.
All code, including pre-trained normative models, tutorials, and analysis scripts are available online and very well documented. In addition, the authors are promising to make an easy-to-use online portal available soon.
Weaknesses
Although still an impressively large multi-site data set, the sample size of the functional data (N=22k) is considerably smaller than that of the structural data (N=58k) which implies higher uncertainty in the functional normative model estimates.The scope of functional normative models computed and shared by the authors is limited to coarse parcellations (based on the Yeo-17 and Smith-10 atlases). High-dimensional functional normative models, for now, still belong to the realm of future work.
Interpretation of deviation scores in classification and prediction tasks is not straightforward. Unlike raw data models, these derived summary measures do not have biological or clinical meaning on their own and can only be interpreted with respect to a pre-defined set of reference data.
-
Reviewer #3 (Public Review):
This important study continues the development of normative models of neuroimaging-derived features initiated by themselves (Rutherford et al., 2022a) in two directions. First, the existing models - which were developed on structural imaging features - are complemented with features derived from functional networks. Second, these models are compared with the utilization of the features themselves in three different inference settings. Overall, the evaluation of the functional networks modeling yielded similar benchmarking metrics in agreement with their previous structural modeling. The study delivers strong evidence that normative models efficiently increased the statistical power in mass univariate group difference testing. The improvement in the other two inferential scenarios was less evident. However, …
Reviewer #3 (Public Review):
This important study continues the development of normative models of neuroimaging-derived features initiated by themselves (Rutherford et al., 2022a) in two directions. First, the existing models - which were developed on structural imaging features - are complemented with features derived from functional networks. Second, these models are compared with the utilization of the features themselves in three different inference settings. Overall, the evaluation of the functional networks modeling yielded similar benchmarking metrics in agreement with their previous structural modeling. The study delivers strong evidence that normative models efficiently increased the statistical power in mass univariate group difference testing. The improvement in the other two inferential scenarios was less evident. However, normative modeling was not comparatively detrimental and should continue to be investigated.
The study showcases several major strengths:
- The methodological approach is robustly supported by previous work and protocol definitions by the authors, mainly (Rutherford, 2022a; 2022b).
- The intent of the manuscript is very clear, structured first with a confirmation of the soundness of their functional-networks model and second the "head-to-head" comparison (a term used in the abstract which effectively describes the aim) to alternative inference approaches.
- The results of task 1 are very compelling. The other two tasks, while perhaps less robust, are definitely relevant to be part of the communication and help draw a more accurate picture of the role of normative models in years to come.
- The manuscript is accompanied by a comprehensive set of tutorials, examples, documentation, and the sharing of code, models, and data. Sharing all these resources is a decisive effort toward research transparency that deserves full recognition as scientific scholarship.As major weaknesses, I will speculate that some researchers could understand this work as incremental. Although there's continuity with the previous work of the authors (otherwise would be a weakness, in my opinion), my assessment is that the science in this manuscript should be considered new results and hence deserve independent communication.
Finally, I would like to highlight how normative modeling outperformed its "direct" (saving the removal of confounding factors) inference counterpart in task 1, providing solid evidence of the usefulness of normative models beyond the classical application in "easy" clinical decisions (I refer the readers to the manuscript, which elaborates on these aspects more appropriately and comprehensively).
-
