Design issues and solutions for stop-signal data from the Adolescent Brain Cognitive Development [ABCD] study

  1. Patrick G. Bissett
  2. McKenzie P. Hagen
  3. Henry M. Jones
  4. Russell A. Poldrack

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

The Adolescent Brain Cognitive Development (ABCD) study is an unprecedented longitudinal neuroimaging sample that tracks the brain development of over 10,000 9-10 year olds through adolescence. At the core of this study are the three tasks that are completed repeatedly within the MRI scanner, one of which is the stop-signal task. In analyzing the available stopping experimental code and data, we identified a set of design issues that we believe significantly compromise its value. These issues include but are not limited to: variable stimulus durations that violate basic assumptions of dominant stopping models, trials in which stimuli are incorrectly not presented, and faulty stop-signal delays. We present eight issues, show their effect on the existing ABCD data, suggest prospective solutions including task changes for future data collection and preliminary computational models, and suggest retrospective solutions for data users who wish to make the most of the existing data.

Article activity feed

  1. eLife

    ###Reviewer #3:

    Bissett and colleagues provide an in-depth assessment of the stop signal task implementation in the ABCD protocol. Given the importance of the data set itself, as well as current trends in research funding, there are several important lessons to be learned here, both regarding this specific task implementation, as well as with respect to task designs in large-scale data collections in general.

  2. eLife

    ###Reviewer #2:

    This paper reports a thorough critique of the ABCD stop-signal data set. It identifies a set of eight problems that severely limits the utility of the ABCD stopping data. In particular, the first two (which are essentially the same problem) invalidate estimates of SSRT based on the independent race model because of violations of the context independence assumption of that model. The remaining issues are more minor in the sense that while potentially problematic they either affect a very small percentage of the data and so can be dealt with by removing the affected trials or participants, or do not appear to be problematic in practice.

    The authors have provided a valuable service to the research community in systematically and thoroughly cataloguing these issues, although we think it is fair to say that a number of people (including the present reviewers) have been aware of the key design issue caused by the stop signal replacing the go signal for quite some time and have been working on solutions.

    Below we have a few suggestions for clarifications, but overall the paper is very clear and well written.

    Although the paper mentions that "new models for stopping must be developed to accommodate context dependence (Bissett et al., 2019), the latter of which we consider to be of utmost importance to advancing the stop-signal literature", it does not discuss such models and neither does it show the potentially severe consequences of context independence violations in the ABCD data set.

    All our more substantive comments relate to "Retroactive Suggestions For Issue 1". First, the authors write: "Given the above, if analyzing or disseminating existing ABCD stopping data, we would recommend caution in drawing any strong conclusions from the stopping data, and any results should be clearly presented with the limitation that the task design encourages context dependence and therefore stopping behavior (e.g., SSRT) and neuroimaging contrasts may be contaminated".

    We feel that this recommendation is too lenient and would suggest the following alternative: Unless the ABCD community conclusively shows that the design flaw does not distort conclusions based on SSRT estimates (or any other stop-signal measure), researchers should not use the ABCD data set to estimate SSRTs at all.

    Second, the authors suggest removing subjects who have severe violations as evidenced by mean stop-failure RT > mean no-stop-signal RT. We are concerned that this recommendation impacts on the representativeness of the sample. Also, this recommendation ignores the fact that violations are not an all-or-none phenomenon but are a matter of degree and can come in varying shapes and sizes.

    Third, the authors recommend that "any results be verified when only longer SSDs are used, perhaps only SSDs > 200ms". Figure 3 does not seem to support the recommended cut-off of 200ms: at 200ms accuracy is still far from asymptotic.

    In general, we feel that recommendations based on removing participants and trials are not sufficient. Such practices will affect the representativeness of the sample and will increase estimation uncertainty and hence decrease power. We believe that the only way to solve Issue 1 is by developing measurement models that can account for the dependence of the go and the stop process.

  3. eLife

    ###Reviewer #1:

    General assessment:

    The paper points out eight design issues observed in the stop signal task of the longitudinal Adolescent Brain Cognitive Development (ABCD) study by Casey et al. (2018). The issues are ordered by importance and are partially interrelated. The paper is written in a very clear and non-redundant style and makes a number of suggestions on how to deal with the various issues. The points made in the paper are well-taken. Moreover, the preprint of this paper has already elicited a reply by authors from the ABCD study leading to some partial adjustments of the design of the stop task.

    Major comments:

    1. As the authors suggest, the most important issue is the potential violation of the context invariance assumption due to the variability of the go stimulus duration across different stop signal delays (SSDs). This is a plausible concern even if the number of "clear" violations is relatively small (447 out of 7231 subjects). Nevertheless, the authors' point would be made even more convincing if they could point to some (simulation?) results showing the effect of a weaker go signal at short SSDs on the estimate of the stop signal response time (SSRT).

    2. I suggest using the term "context invariance" instead of "context independence" , in order not to confound the assumptions of 'context' and 'stochastic' independence in the Logan-Cowan race model. It should be pointed out that the prediction of the race model concerning faster stop failures than go responses is conditional on both context invariance AND stochastic independence between go and stop signal processing being true (see Colonius & Diederich, 2018, Psych. Review).

    3. I have no further major comments but would like to suggest a further analysis: Let us suppose, as the authors point out, that the RT distribution of responses to the go signal is indeed affected by the duration of the go signal. As a first approximation, let us assume that the observed RT distribution is a binary mixture of responses: slow RTs to a weak/short go stimulus and fast RTs to a strong/long gos stimulus. Without making specific assumptions about the two components of the mixture, one could employ a mixture distribution test first suggested by Falmagne (1968, British J. Math. Statist. Psychology): The RT ("density") distributions, plotted separately for each SSD and go signal trials, should all cross at one and the same point in time. Of course, this is not a foolproof test but if some evidence in favor of this prediction is found it would strengthen the authors' point.

  4. eLife

    ##Preprint Review

    This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to version 4 of the manuscript. Birte Forstmann (University of Amsterdam) served as the Reviewing Editor.

    ###Summary:

    This paper focuses on one of the benchmark magnetic resonance imaging (MRI) datasets, the so-called Adolescent Brain Cognitive Development (ABCD). In total, eight design issues observed in the stop signal task of the longitudinal ABCD study by Casey et al. (2018) are pointed out. The design issues are described in detail, ordered by importance, and a number of suggestions are given on how to overcome potential limitations. Given the importance and prominence of the ABCD study in the field of cognitive neurosciences, both the reviewers and editors believe this paper to highlight essential issues in a constructive way. Finally, we believe this paper will elicit a fruitful discussion including the adjustments of the design of the stop signal task.

    Overall, this manuscript is well written, interesting, timely and will help resolve the debate in the field. We have the following suggestions to improve the manuscript.

  5. Version published to 10.1101/2020.05.08.084707 on bioRxiv