Reproducible functional connectivity endophenotype confers high risk of ASD diagnosis in a subset of individuals

  1. Sebastian GW. Urchs
  2. Hien Duy Nguyen
  3. Clara Moreau
  4. Christian Dansereau
  5. Angela Tam
  6. Alan C. Evans
  7. Pierre Bellec

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Functional connectivity (FC) analyses of individuals with autism spectrum disorder (ASD) have established robust alterations of brain connectivity at the group level. Yet, the translation of these imaging findings into robust markers of individual risk is hampered by the extensive heterogeneity among ASD individuals. Here, we report an FC endophenotype that confers a greater than 7-fold risk increase of ASD diagnosis, yet is still identified in an estimated 1 in 200 individuals in the general population. By focusing on a subset of individuals with ASD and highly predictive FC alterations, we achieved a greater than 3-fold increase in risk over previous predictive models. The identified FC risk endophenotype was characterized by underconnectivity of transmodal brain networks and generalized to independent data. Our results demonstrate the ability of a highly targeted prediction model to meaningfully decompose part of the heterogeneity of the autism spectrum. The identified FC signature may help better delineate the multitude of etiological pathways and behavioural symptoms that challenge our understanding of the autism spectrum.

Article activity feed

  1. eLife

    ###Reviewer #3:

    In this manuscript, Urchs and colleagues use transductive conformal prediction (TCP) applied to rsfMRI functional connectivity data to predict autism in a subset of cases. The approach is novel for applying to autism research and also is pinpointed at a topic that is very much needed in autism - the problem of heterogeneity. The logic applied is that only a subset of autism cases will have powerful biomarker differences in terms of resting state functional connectivity and TCP is utilized to isolate that subset. Thus, while the approach is novel and maps onto similar kinds of logic in the realm of genetics of autism, the utility is somewhat limited, as TCP will not be able to tell us much about the majority of cases. This is the same problem with many highly penetrant genetic mechanisms that lead to high risk for autism. However, it is still an issue that the approach can only make statements about a very small percentage of the total autism cases in the population. Could the authors comment more on this issue/limitation? For instance, what does this biomarker in a small percentage of cases tell us? Are there powerful, specific, and homogeneous biological mechanisms behind such cases, whereas for the rest of the population the underlying mechanisms are highly diverse and not powerful enough to penetrate up into macroscale functional connectivity phenotypes? The result could help to generate new hypotheses focused on such a group. However, I think the authors should try to lead readers in discussing how to take such results further for new discoveries.

    Besides this main issue noted above about the utility or meaning behind the novel findings, the following are comments about how to make the introduction more readable, and how to potentially better facilitate a reader's understanding of the analyses.

    1. Introduction: I would suggest that some modifications need to be done to the introduction in order to make the ideas flow a bit better. The problem is that the authors are introducing a variety of complex and not necessarily easily linked information - e.g., risk from a variety of different types of genetic mechanisms, failure of neuroimaging classifier studies, and TCP. With a bit of effort and a couple re-readings it is clear that the logic the authors are using is that we have some understanding of how much risk there is from different types of genetic mechanisms, and we would like to understand how neuroimaging data might match up to that. Using TCP would hopefully allow you to do that, hence the goals of the study. This logic is not clearly spelled out as one reads the introduction however, because the different topics are either mixed together within a paragraph with little linking text to help the reader follow the logic, or the bits of information for each topic are segregated into their own paragraphs with little linking text and the beginning or ends of the paragraphs to help the ideas flow from one paragraph to the next. A good example of this is that the background paragraph to start with has these topics mixed together within the very first paragraph, and then the subsequent 3 paragraphs solely focus on each topic, without helping the reader understand why they are jumping from very different topics. By the time the reader gets to line 120 of the Objectives, then things are spelled out a little better, but the reader has to then go back and connect the ideas about how the authors are trying to compare how a TCP approach to identify a high risk imaging marker would match up against more well known risk markers at the genetic level. It may be the case that the manuscript here will get readers of various different backgrounds (e.g., autism researchers, those with expertise in genetics, neuroimaging, or machine learning). Few have expertise in all those areas, and for those individuals, it may be hard to understand how these different topics flow together and are linked in a specific logical way. The logic is there, but even for this reviewer, it required a couple readers to see how all this information lined up in a logic way to justify the study. Thus, I would suggest that the authors make changes to the writing so that the reader can clearly follow the logic without too much extra effort to connect what isn't written about how these topics are supposed to line up.

    2. Methods: The methods and analysis are fairly complex. Can the authors make a figure that clearly lays out the analysis pipeline? It would help to have a visual that clearly outlines how the authors selected the subset of individuals from the larger ABIDE datasets, how the preprocessing was done, how the features were estimated, and how the TCP analysis was implemented with all the associated added aspects like the bootstrapping, etc. Furthermore, to facilitate understanding of the complexities of the analysis, can the authors create a GitHub repo that has all the reproducible analysis code that generates the results and figures produced in the paper, along with tidy data files that have the features used by the TCP model? Although in the data availability statement the authors write that a GitHub repo exists, having had a look through this, no tidy data files are available that the code can load up to have readers reproduce the analysis or figures. In addition, the code consists of only 4 brief R scripts. That code isn't easily readable with regards to how the analysis was done. The R code could be done in another way that is more in line with literate programming, such as an Rmd file, that has the analysis code, along with plain text to describe the different steps, and then the figures embedded within the html or pdf report that it creates when it is knitted in R Studio. There are also some Jupyter notebooks that show how the figures were generated. This was helpful to see and is what is needed for the R code too. In those Jupyter notebooks, it seems like there are certain tidy data files that those notebooks load, but they are absent in the repository and therefore, the readers cannot reproduce the analysis.

  2. eLife

    ###Reviewer #2:

    This work represents an investigation into autism(s). For this purpose, multi-network inputs to transductive conformal prediction are used. This approach provides a measure for how much an individual resembles a pattern linked to autism(s) or healthy controls. The resulting predictions are translated to the population prevalence. The authors state correctly that their models are in the ballpark of what has previously been reported. However, they claim that their improvements with respect to predictions in the general population are a major improvement, achieved by a bias towards specificity of their model. While machine learning papers often do not report this translation it is also apparent that they easily could. Therefore, the novelty of this approach is not clear to me as it may be to the authors. This requires clarification in the context of the literature in addition to addressing the major concerns below.

    1. The paper would benefit from a more in depth discussion of the literature. There have been more than 50 papers published using different pattern recognition approaches on ASD. It is important that the authors evaluate their work in the context of those findings. There are a bunch of reviews on pattern classification approaches in psychiatry in general and ASD in particular.

    2. A slightly longer and more in-depth description of the methods section would help the reader, especially a description of the method used to calculate the relevant score.

    3. Based on Figure 3 it is a bit unclear to me if the small number of individuals identified with higher HRS score indeed also show higher symptoms. This should be statistically tested.

    4. The strongest confounding effects are usually induced by scanner differences, as both the discovery as well as the replication sample are multi-site samples. It would be important to investigate the effect of scanners on the proposed models. This is particularly problematic should there be disbalances between the groups across scanners.

    5. Probabilistic predictive approaches have already been applied to ASD using for instance gaussian process regression (e.g. Ecker et al. 2010, Neuroimage). The paper would benefit by stating clearly how their method improves above the approach mentioned in this referred paper as well as other approaches in ASD. The adjustments of the prediction to the population prevalence is a minor achievement.

    6. The authors discuss: "Although our model made only few predictions, those predictions carry a much higher risk of an ASD diagnosis for the identified individuals. The result is a prediction with a much higher specificity (99.5% compared to 72.3% and 63% for traditional approaches, Heinsfeld et al., 2018; Abraham et al., 2017) and much lower sensitivity (4.2%, compared to 61% and 74% respectively). It is thus important to point out that here we have not proposed a better prediction learning model, but rather addressed a different objective." However, sensitivity and specificity are always a trade-off and dependent on the decision threshold. You can bias this for either of the two. For probabilistic models this is easy to do by adjusting the decision threshold to the population prevalence of a disorder. It is also possible to determine a decision margin which will naturally lead to higher performance, similar to the approach presented here and has been done and proposed earlier.

  3. eLife

    ###Reviewer #1:

    This is a well-written manuscript examining prediction of ASD diagnosis from resting-state fMRI data. The primary innovation is the application of Transductive Conformal Prediction (TCP), which quantifies the confidence with which one can accurately make a prediction. The authors show that they can identify a functional connectivity (FC) signature with high PPV for a subset of patients.

    The approach is certainly interesting, but it also seems circular. As I understand it, predictions are limited only to individuals who can be classified with high accuracy. A priori, we might expect that these people would be patients with severe illness, and the results show that the subset of patients who are correctly identified do have more severe symptoms. It therefore seems unfair to compare the high PPV of this method with other approaches, when the current method, by construction, focuses only on those cases who are easier to classify (whereas others don't). Could the authors please clarify whether this interpretation is accurate?

    Related to the above, the PPV of the test is high, but this is only one side of the coin. The sensitivity is very low and I imagine the NPV is also low. Given its low sensitivity, It does not seem correct to speak of the FC signature as a risk marker, since many people at risk (indeed with a diagnosis) do not show it. In practical terms, it seems like a positive result with this FC marker is conservative, relatively accurate indicator of someone's risk for a severe form of ASD, but a negative result carries almost no information at all. What is the practical utility of such a marker, given that severe autism should be evident from clinical observation? That is, how could the current results add value to clinical decision-making? If the FC signature could be detected in newborns, it would be of value, but this analysis is conducted in adults after diagnosis has been established.

    The methods section indicates that the approach prioritises specificity, but the reasons for this decision are unclear.

    How were site differences addressed in the analysis?

    It would be useful to see how results vary as the 5% threshold is varied.

    The evidence for cluster structure in Fig 1b seems quite weak.

    The Figure 1 caption requires greater detail explaining what is actually shown in the plots.

    Were any of the participants taking psychotropic medications? to what extent could this have impacted the findings?

  4. eLife

    ##Preprint Review

    This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to version 1 of the manuscript.

    ###Summary:

    The reviewers shared a number of concerns in common, as outlined in their detailed reviews. In addition, the following points were raised upon further discussion between the reviewers:

    -A comprehensive analysis of the potentially confounding effect of site differences is required

    -The potential circularity of the method - classifying only cases that can be confidently classified - and practical limitations of this approach should be discussed in greater detail. The algorithm is biased towards specificity. This could also be achieved using probabilistic machine learning approaches by, for instance, adjusting the decision threshold to the population prevalence or by defining a margin for cases for which you do not make a decision.

    -The findings are considered in relation to population prevalence rates, but the algorithm is not applied to a population sample. It seems likely that the classifier would not detect cases with the same accuracy in a population sample. If this claim is made, it needs to be explicitly tested.

    -The passage "The result is a prediction with a much higher specificity (99.5% compared to 72.3% and 63% for traditional approaches, Heinsfeld et al., 2018; Abraham et al., 2017) and much lower sensitivity (4.2%, compared to 61% and 74% respectively)." seems problematic. If you calculate the balanced accuracy for the current approach, of Specificity + Sensitivity/2, you end up slightly above chance accuracy. The other papers actually perform better.

  5. Version published to 10.1101/2020.06.01.127688 on bioRxiv