Normative age modelling of cortical thickness in autistic males

Article activity feed

1. 

   eLife

   Feb 20, 2020

   ###Reviewer #2:

   I very much like the general idea of this paper, but my opinion is that this is not an idea that can/should be applied to these data. As elaborated below, the ABIDE data are from numerous sites with different scanners, imaging acquisition sequences and parameters, sample ascertainment, etc, The methods used in the current paper rely on there not being such heterogeneity; and its presence can either render true ASD-related deviance invisible, or create an illusion of ASD-related deviance where there is none. Such heterogeneity is, of course, problematic for more conventional approaches; but is far more problematic for the methods proposed here.

   Major Issues and Questions:

   1. The authors are critical of case-control models but do not present an alternative to dealing with the heterogeneity in the data. Indeed, linear models …

   ###Reviewer #2:

   I very much like the general idea of this paper, but my opinion is that this is not an idea that can/should be applied to these data. As elaborated below, the ABIDE data are from numerous sites with different scanners, imaging acquisition sequences and parameters, sample ascertainment, etc, The methods used in the current paper rely on there not being such heterogeneity; and its presence can either render true ASD-related deviance invisible, or create an illusion of ASD-related deviance where there is none. Such heterogeneity is, of course, problematic for more conventional approaches; but is far more problematic for the methods proposed here.

   Major Issues and Questions:

   1. The authors are critical of case-control models but do not present an alternative to dealing with the heterogeneity in the data. Indeed, linear models are inadequate to deal with the heterogeneity in the ABIDE data given the lack of overlap in the data for different sites. But the normative approach presented here seems to not deal with the problem at all, potentially transforming what would be taken out by a nuisance variable into an alteration in ASD-related deviance.

   2. The sparsity of the data beyond childhood is extremely problematic for this approach. The approach of taking data in one-year bins requires large amounts of data within each bin to make the means and standard deviations reliable. By the teenage years, this is clearly not the case. The authors limit age bins to having at least 3 control points; this is clearly wildly insufficient, and would be even if there were no issues with site heterogeneity. Conventional linear models are to be preferred to normative models under these conditions.

   3. The comparison of results from a case-control model versus a normative model seems misleading. A case-control model approach requires a specification of the age at which the comparison is made. This is not provided, leading one to suspect that the age data were not centered, but were absolute, and thus the differences were essentially projecting backwards to birth. (This is, I believe, a common mistake.) The model specification is also completely lacking. Moreover, a case-control approach does not preclude the possibility of centering the data at different ages (as in e.g. Khundrakpam et al. (2017)). Between this and the problems with heterogeneity for the normative models, it is unclear how to interpret these results.

   4. The idea that individuals that are more than 2 stddevs away from the mean of the controls are outliers and should be eliminated from the analysis seems mistaken. If all individuals with ASD are substantially far from the mean of the controls, they are clearly not to be treated as outliers.

   5. The impact statement claims that "normative modelling has the potential to isolate specific highly deviant subsets of individuals with ASD, which will have implications for understanding the underlying mechanisms and bring clinical impact closer"; there is no indication that that is the case. The normative model has identified primarily children, and has identified nothing in particular about those children. Case-control models have done the same.

   6. It appears to this reviewer that this paper outlines an approach which could be worthwhile in a data set without massive heterogeneity, but within the context of the ABIDE data actually seems harmful.

   Read the original source
2. 

   eLife

   Feb 20, 2020

   ###Reviewer #1:

   This paper describes the impact of outliers in normative cortical thickness (CT) measurements when examining those suffering from autism spectrum disorder (ASD). The authors used the ABIDE sample and binned subjects by age, and assessed outliers as a function of a "w-score" which they estimated across CT parcellations across the entire cortex. They then demonstrate that cortical thickness differences that can ascribed to ASD can essentially be attributed to a small number of outliers within the sample. They also demonstrate that this w-score may be sensitive to clinical variables as well.

   Overall, it is unclear to me what the exact goal of the work is: To describe the anatomy of ASD better? To subtype? Or is there another "take-home" message of this paper? I would imagine that the case-control differences in most …

   ###Reviewer #1:

   This paper describes the impact of outliers in normative cortical thickness (CT) measurements when examining those suffering from autism spectrum disorder (ASD). The authors used the ABIDE sample and binned subjects by age, and assessed outliers as a function of a "w-score" which they estimated across CT parcellations across the entire cortex. They then demonstrate that cortical thickness differences that can ascribed to ASD can essentially be attributed to a small number of outliers within the sample. They also demonstrate that this w-score may be sensitive to clinical variables as well.

   Overall, it is unclear to me what the exact goal of the work is: To describe the anatomy of ASD better? To subtype? Or is there another "take-home" message of this paper? I would imagine that the case-control differences in most neurodevelopmental disorders with high heterogeneity and high variability would demonstrate a similar kind of trend. And thus, at the end of the day, I am not sure how much this technique advanced our understanding of ASD.

   Issues and Questions:

   1. It is unclear from the methods how the authors deal with motion and image quality. Recent work by Pardoe and Bedford demonstrate the importance of dealing with this issue, particularly in the context of the ABIDE sample. This would likely have a significant impact on any of the results. It's unclear if the use of the Euler index at the extremes of the distribution of the dataset being used is sufficient. How did the authors come up with their Euler number cut-off?

   2. The W-score could use a much better explanation. It is not clear to me as to what it is and how this should be interpreted. The lack of information regarding the number of age-bins used also makes interpreting these findings confusing in my mind.

   3. The authors report that, "The median number of brain regions per subject with a significant p-value was 1 (out of 308), indicating that the w-score provides a robust measure of atypicality." I guess this could be true, but given the variation in normative ageing and development, I suspect this would also be true of a large number of TD children. That being the case, would it be worth doing a permutation test to determine the threshold of how man "atypical" areas one could expect by chance?

   4. The authors note "Unfortunately, despite a significant female subgroup, the age-wise binning greatly reduced the number of bins with enough data-points in the female group." I understand that this could indeed be a problem. However, I think it would be good for the authors to provide more details. Potentially a histogram to demonstrate the issue. My feeling is that with sex difference with respect to ASD, the more information that could be provided the better. Overall, it is unclear to me as to how useful a sex-specific analysis may be in this particular context given the sample sizes available in ABIDE.

   5. Results, page 8: "Because we also had computed w-scores from our normative age-modelling approach, we identified specific 'statistical outlier' patients for each individual region with w-scores > 2 standard deviations from typical norms and excluded them from the case-control analysis."

   I'm not sure I agree with the premise of this statement. First, it is hard to know without seeing all of the data, but based on Fig 1, it seems that there are ASD individuals that fall on both sides of this distribution. So if there are effect sizes that can be gleaned, this would be in spite of the variability. Second, it would be paramount to determine how many people are outliers-by-region. This, in and of itself, would be useful information. If a significant proportion of individuals can be identified as outliers, this suggests that variability is the norm rather than an exception. I'm skeptical as to whether you get interesting information from removing these individuals from analyses.

   1. Result, page 9: "While the normative modelling approach can be sensitive to different pathology." I don't think you're capturing anything interesting about pathology with this method, especially as it pertains to CT values.

   2. Result, page 9-10: I'm still confused by this notion of atypicality. Presumably this suggests that 5-10% of all ASDs are more than 2SDs from a normative distribution. But is this at both tails of the distribution? There are significant interpretational issues with this. thus, it is imperative on the authors to do a better job of describing these distributions.

   3. Part of the rationale of this paper is that using the w-score is far more robust than using simple CT values. I'm sure that residualized CT values could have been used for any of these analyses. If that were to be done how would this change the results?

   Minor comments and suggestions on presentation:

   1. While this paper has some merits, I found it hard to read. There is not a clear delineation between the methods and the results, and some methodological considerations are written into the results section and vice-versa.

   2. In the introduction, the authors use the word "deviance" to describe what appears more to me like age-related variation and heterogeneity in ASD. Deviance may be too strong a term and easily mis-interpretable. I would suggest replacing it with something a bit more like variation. Also, the work at the institution of the main author (for example by Baron-Cohen and authors) really champions the use of terms like "neurotypical" rather normally developing. I think, in general, the authors may want to take their cues from this type of language.

   3. This passage in the Introduction need of references. The work by Hong (in Boris Bernhardt's group), Bedford (in Mallar Chakravarty's group), Schuetze (in Signe Bray's group), and Meng-Chuan Lai all come to mind.

   "Even within mesoscopic levels of analysis such as examining brain endophenotypes, heterogeneity is the rule rather than the exception (Ecker, 2017). At the level of structural brain variation, neuroimaging studies have identified various neuroanatomical features that might help identify individuals with autism or reveal elements of a common underlying biology (Ecker, 2017). However, the vast neuroimaging literature is also considerably inconsistent, with reports of hypo- or hyper-connectivity, cortical thinning versus increased grey or white matter, brain overgrowth, arrested growth, etc., leaving stunted progress towards understanding mechanisms driving cortical pathophysiology in ASD."

   1. I found the Discussion missed the mark. It was mostly written as a rehash of the results, with no real biological interpretation. There is not a sufficient examination of the relationship of these findings to other important papers (Kundrakpham, Bedford, Hong, Ecker, Hyde, Lange, etc...).

   2. Figure 3 - The colour bars should be labelled.

   Read the original source
3. 

   eLife

   Feb 20, 2020

   ##Preprint Review

   This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to Version 4 of the preprint: https://www.biorxiv.org/content/10.1101/252593v4

   ###Summary:

   This paper uses data from the Autism Brain Imaging Data Exchange (ABIDE) to model the relationship between cortical thickness in different brain regions and patients with autism spectrum disorders (ASD) compared to neurotypical controls. The reviewers appreciated the goals and approach of this paper, but, as described below, had questions about the suitability of the data for this analysis, the ways in which the data were processed, the way in which the results …

   ##Preprint Review

   This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to Version 4 of the preprint: https://www.biorxiv.org/content/10.1101/252593v4

   ###Summary:

   This paper uses data from the Autism Brain Imaging Data Exchange (ABIDE) to model the relationship between cortical thickness in different brain regions and patients with autism spectrum disorders (ASD) compared to neurotypical controls. The reviewers appreciated the goals and approach of this paper, but, as described below, had questions about the suitability of the data for this analysis, the ways in which the data were processed, the way in which the results were interpreted, and the significance of these findings for understanding autism spectrum disorders.

   Read the original source
4. 

   Version published to 10.1101/252593 on bioRxiv

   Jan 23, 2018