Bayesian model selection favors parametric over categorical fMRI subsequent memory models in young and older adults
This article has been Reviewed by the following groups
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
- Evaluated articles (eLife)
Abstract
Subsequent memory paradigms allow to identify neural correlates of successful encoding by separating brain responses as a function of memory performance during later retrieval. In functional magnetic resonance imaging (fMRI), the paradigm typically elicits activations of medial temporal lobe, prefrontal and parietal cortical structures in young, healthy participants. This categorical approach is, however, limited by insufficient memory performance in older and particularly memory-impaired individuals. A parametric modulation of encoding-related activations with memory confidence could overcome this limitation. Here, we applied cross-validated Bayesian model selection (cvBMS) for first-level fMRI models to a visual subsequent memory paradigm in young (18-35 years) and older (51-80 years) adults. Nested cvBMS revealed that parametric models, especially with non-linear transformations of memory confidence ratings, outperformed categorical models in explaining the fMRI signal variance during encoding. We thereby provide a framework for improving the modeling of encoding-related activations and for applying subsequent memory paradigms to memory-impaired individuals.
Article activity feed
-
###Reviewer #3:
Summary of the manuscript:
This manuscript carefully explores different ways of analyzing fMRI data acquired during a subsequent memory paradigm. Subsequent memory paradigms (and variants thereof) are widely used in human memory research. The paradigm involves assessing activity-dependent encoding by first presenting novel stimuli (typically during human brain imaging), before classifying the stimuli post hoc using behavioral performance on a subsequent recognition test. Here, the authors use a subsequent memory paradigm to collect fMRI data from 256 volunteers, including both young (<35 years old) and older populations (>50 years old). The authors then perform cross-validated Bayesian model selection to compare categorical and parametric approaches to data analysis. The authors show that parametric models (particularly …
###Reviewer #3:
Summary of the manuscript:
This manuscript carefully explores different ways of analyzing fMRI data acquired during a subsequent memory paradigm. Subsequent memory paradigms (and variants thereof) are widely used in human memory research. The paradigm involves assessing activity-dependent encoding by first presenting novel stimuli (typically during human brain imaging), before classifying the stimuli post hoc using behavioral performance on a subsequent recognition test. Here, the authors use a subsequent memory paradigm to collect fMRI data from 256 volunteers, including both young (<35 years old) and older populations (>50 years old). The authors then perform cross-validated Bayesian model selection to compare categorical and parametric approaches to data analysis. The authors show that parametric models (particularly those with non-linear transformations) out-perform categorical models in explaining the fMRI signal variance during encoding.
General assessment:
The strengths of this manuscript are two-fold. First, the authors illustrate application of a recently published SPM toolbox (Soch et al., 2016; Soch and Allefeld, 2018), used to conduct model assessment, comparison and selection. Second, the manuscript shows that parametric models out-perform categorical models when applied to subsequent memory paradigms. The manuscript is methodologically rigorous and illustrates a pipeline for optimizing GLMs applied to fMRI data. It uses data from a large number of subjects and results are replicated in an independent cohort. The manuscript will provide a useful reference for those researchers designing subsequent memory paradigms or performing analyses on data deriving from this particular paradigm.
Having said this, by focusing on methodological questions relating specifically to subsequent memory paradigms, the manuscript is relatively narrow in scope. Moreover, despite providing the first formal comparison of categorical and parametric models for data acquired from subsequent memory paradigms, researchers have been applying both types of model to data deriving from this task for more than 10 years.
Major comments:
The authors do not present behavioral results, yet it seems the variance in confidence on the recognition test underlies the success of the parametric modeling approach. Moreover, it seems important to show whether there are any behavioral differences between young and old adults, given the framing of the Introduction where the authors note that categorical modeling approaches may be limited by ceiling effects in young populations and low accuracy in older populations. Using the behavioral data alone, can the authors illustrate these limitations of the categorical approach?
In the Introduction the authors emphasize the importance of their approach for identifying biomarkers that predict normal aging versus accelerated aging in humans. Given this comparison is not made, it seems more appropriate to move this section of the Introduction to the Discussion?
Clarity of the Results section: The results are somewhat dense and hard to follow at times. One notable factor is the lack of clarity in the figures, where the key point conveyed by each figure is not always immediately apparent. Here are some suggestions to help improve this section of the manuscript:
a) Figure 3, Figure 4A, Figure 5, Figure 6, Figure 8: it is difficult to distinguish between the red/blue/magenta colours. Can the authors use 3 colours that are more different?
b) Can the authors explicitly state what they expect to see on selected-model maps? Given the main audience for this manuscript will be from the fMRI community, it is important that these maps are not confused with maps showing task-related modulation of the BOLD signal.
c) Can the authors describe in more general terms the rationale behind all the different categorical models? By considering so many different models I wonder if the key comparison between categorical and parametric gets lost in the detail.
d) Figure 3: I'm not sure how helpful this figure is for the main Results section? It doesn't address the key question posed by the authors, so is it not more suitable for the Supplement?
e) How representative are the plots shown in Figure 4B? Do the authors observe the same gradient if assessing log Bayes factor in an ROI defined from previous subsequent memory paradigms?
f) Section 4.2. It isn't immediately clear why models that do not include subsequent memory effects are included, if the key comparison is between subsequent memory effects in categorical and parametric models.
g) Figure 5: The authors distinguish between 'theoretical' and 'empirical' parametric modulators. If both are defined using behavioural performance, then what is the rationale for these terms?
-
###Reviewer #2:
This paper describes efforts to evaluate and compare different models of a subsequent memory paradigm. In particular, the goal is to improve sensitivity so that the paradigm can be used more effectively in older adults who may have memory problems.
The paper is well written overall, and the sample size is impressive. I also think that improving sensitivity to detect memory deficits during aging and disease progression is an important goal. Finally, the approach is rigorous, as cvBMS provides a principled means of model comparison and validating the findings in another cohort is very laudable.
That said, the paper is overly focused on a specific paradigm and it does not provide insights into neural underpinnings of a biological/cognitive function. To be clear, the goal of the paper does not appear to be to provide such …
###Reviewer #2:
This paper describes efforts to evaluate and compare different models of a subsequent memory paradigm. In particular, the goal is to improve sensitivity so that the paradigm can be used more effectively in older adults who may have memory problems.
The paper is well written overall, and the sample size is impressive. I also think that improving sensitivity to detect memory deficits during aging and disease progression is an important goal. Finally, the approach is rigorous, as cvBMS provides a principled means of model comparison and validating the findings in another cohort is very laudable.
That said, the paper is overly focused on a specific paradigm and it does not provide insights into neural underpinnings of a biological/cognitive function. To be clear, the goal of the paper does not appear to be to provide such insights, and is instead to "...identify several ways to improve the modeling of subsequent memory effects in fMRI".
-
###Reviewer #1:
General assessment:
The topic discussed in the current manuscript is interesting and the proposed framework will be a great addition to the traditional methods currently used in the studies of human memory. The manuscript investigated the applicability of parametric compared to categorical models of subsequent memory effects in fMRI. Specifically, the authors applied cross-validated Bayesian model selection (cvBMS) for fMRI models to a subsequent memory paradigm in young and older adults. The cvMBS results showed that parametric models better explained the encoding signals when compared to categorical counterparts, suggesting a new analytical framework that can be applied to participants with low memory performance including memory-impaired individuals whose data would otherwise be challenging to interpret.
Major comments:
###Reviewer #1:
General assessment:
The topic discussed in the current manuscript is interesting and the proposed framework will be a great addition to the traditional methods currently used in the studies of human memory. The manuscript investigated the applicability of parametric compared to categorical models of subsequent memory effects in fMRI. Specifically, the authors applied cross-validated Bayesian model selection (cvBMS) for fMRI models to a subsequent memory paradigm in young and older adults. The cvMBS results showed that parametric models better explained the encoding signals when compared to categorical counterparts, suggesting a new analytical framework that can be applied to participants with low memory performance including memory-impaired individuals whose data would otherwise be challenging to interpret.
Major comments:
Given that the parametric models are a critical part of this manuscript, the rationale and justifications for the use of these models especially in the context of memory fMRI experiments are currently not sufficiently discussed. For example, in the introduction, there is no reference of past findings that are in line with the assumption that BOLD signals in memory-related brain regions vary quantitatively (rather than qualitatively) as a function of the strength of encoding signals. I believe this to be critical in convincing readers why parametric models can and should be used when thinking about memory fMRI data and paradigms.
While the results section is clearly written, I find the analysis section to be rather difficult to follow. Is it possible at all to even more carefully walk through each of the model subtypes with more details or consider setting up a consistent structure for how each model subtype is explained (across model types; i.e., across 3.1, 3.2, and 3.3). In addition, I believe the readers could also benefit from more explanations/motivations behind why certain models should be considered and how to conceptually think about them (e.g., what are some empirical findings which suggest that model GLM with parametric modulators that are linear, arcsine, and sine should be considered here and are good candidates but not others?).
-
##Preprint Review
This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to version 3 of the manuscript.
###Summary:
We found that this paper is of interest to an audience of cognitive neuroscientists who perform subsequent memory experiments. It provides important technical advice for the analysis of this data. The paper is also of interest for researchers who want to carry out similar technical evaluations in other experiments.
Whilst we have some comments that could improve the manuscript, we find the key claims of the manuscript to be well supported by the data, and that researchers who use this paradigm would benefit …
##Preprint Review
This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to version 3 of the manuscript.
###Summary:
We found that this paper is of interest to an audience of cognitive neuroscientists who perform subsequent memory experiments. It provides important technical advice for the analysis of this data. The paper is also of interest for researchers who want to carry out similar technical evaluations in other experiments.
Whilst we have some comments that could improve the manuscript, we find the key claims of the manuscript to be well supported by the data, and that researchers who use this paradigm would benefit from following the advice to use parametric models. Furthermore the approaches used to support these claims are both thoughtful and rigorous.
-
