Hierarchical cortical entrainment orchestrates the multisensory processing of biological motion
Curation statements for this article:-
Curated by eLife
eLife assessment
Wang et al. presented visual (dot) motion and/or the sound of a walking person and found that EEG activity tracks the step rhythm, as well as the gait (2-step cycle) rhythm, with tentative demonstration that the gait rhythm is tracked superadditively (power for A+V condition is higher than the sum of the A-only and V-only condition). The findings will be of wide interest to those examining biological motion perception and oscillatory processes more broadly, with the potential to be important. However, at present, due to some analysis concerns - most notably, evidence of double-dipping for one of the core findings - the evidence is incomplete. Furthermore, some of the theoretical interpretations concerning entrainment must remain speculative when the authors cannot dissociate evoked responses from entrained oscillatory effects.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
Abstract
When observing others’ behaviors, we continuously integrate their movements with the corresponding sounds to achieve efficient perception and develop adaptive responses. However, how human brains integrate these complex audiovisual cues based on their natural temporal correspondence remains unknown. Using electroencephalogram, we demonstrated that cortical oscillations entrained to hierarchical rhythmic structures in audiovisually congruent human walking movements and footstep sounds. Remarkably, the entrainment effects at different time scales exhibit distinct modes of multisensory integration, i.e., an additive integration effect at a basic-level integration window (step-cycle) and a super-additive multisensory enhancement at a higher-order temporal integration window (gait-cycle). Moreover, only the cortical tracking of higher-order rhythmic structures is specialized for the multisensory integration of human motion signals and correlates with individuals’ autistic traits, suggesting its functional relevance to biological motion perception and social cognition. These findings unveil the multifaceted roles of entrained cortical activity in the multisensory perception of human motion, shedding light on how hierarchical cortical entrainment orchestrates the processing of complex, rhythmic stimuli in natural contexts.
Article activity feed
-
eLife assessment
Wang et al. presented visual (dot) motion and/or the sound of a walking person and found that EEG activity tracks the step rhythm, as well as the gait (2-step cycle) rhythm, with tentative demonstration that the gait rhythm is tracked superadditively (power for A+V condition is higher than the sum of the A-only and V-only condition). The findings will be of wide interest to those examining biological motion perception and oscillatory processes more broadly, with the potential to be important. However, at present, due to some analysis concerns - most notably, evidence of double-dipping for one of the core findings - the evidence is incomplete. Furthermore, some of the theoretical interpretations concerning entrainment must remain speculative when the authors cannot dissociate evoked responses from entrained oscillatory …
eLife assessment
Wang et al. presented visual (dot) motion and/or the sound of a walking person and found that EEG activity tracks the step rhythm, as well as the gait (2-step cycle) rhythm, with tentative demonstration that the gait rhythm is tracked superadditively (power for A+V condition is higher than the sum of the A-only and V-only condition). The findings will be of wide interest to those examining biological motion perception and oscillatory processes more broadly, with the potential to be important. However, at present, due to some analysis concerns - most notably, evidence of double-dipping for one of the core findings - the evidence is incomplete. Furthermore, some of the theoretical interpretations concerning entrainment must remain speculative when the authors cannot dissociate evoked responses from entrained oscillatory effects.
-
Reviewer #1 (Public Review):
Summary:
Shen et al. conducted three experiments to study the cortical tracking of the natural rhythms involved in biological motion (BM), and whether these involve audiovisual integration (AVI). They presented participants with visual (dot) motion and/or the sound of a walking person. They found that EEG activity tracks the step rhythm, as well as the gait (2-step cycle) rhythm. The gait rhythm specifically is tracked superadditively (power for A+V condition is higher than the sum of the A-only and V-only condition, Experiments 1a/b), which is independent of the specific step frequency (Experiment 1b). Furthermore, audiovisual integration during tracking of gait was specific to BM, as it was absent (that is, the audiovisual congruency effect) when the walking dot motion was vertically inverted (Experiment …
Reviewer #1 (Public Review):
Summary:
Shen et al. conducted three experiments to study the cortical tracking of the natural rhythms involved in biological motion (BM), and whether these involve audiovisual integration (AVI). They presented participants with visual (dot) motion and/or the sound of a walking person. They found that EEG activity tracks the step rhythm, as well as the gait (2-step cycle) rhythm. The gait rhythm specifically is tracked superadditively (power for A+V condition is higher than the sum of the A-only and V-only condition, Experiments 1a/b), which is independent of the specific step frequency (Experiment 1b). Furthermore, audiovisual integration during tracking of gait was specific to BM, as it was absent (that is, the audiovisual congruency effect) when the walking dot motion was vertically inverted (Experiment 2). Finally, the study shows that an individual's autistic traits are negatively correlated with the BM-AVI congruency effect.
Strengths:
The three experiments are well designed and the various conditions are well controlled. The rationale of the study is clear, and the manuscript is pleasant to read. The analysis choices are easy to follow, and mostly appropriate.
Weaknesses:
I only have one potential worry. The analysis for gait tracking (1 Hz) in Experiment 2 (Figures 3a/b) starts by computing a congruency effect (A/V stimulation congruent (same frequency) versus A/V incongruent (V at 1 Hz, A at either 0.6 or 1.4 Hz), separately for the Upright and Inverted conditions. Then, this congruency effect is contrasted between Upright and Inverted, in essence computing an interaction score (Congruent/Incongruent X Upright/Inverted). Then, the channels in which this interaction score is significant (by cluster-based permutation test; Figure 3a) are subselected for further analysis. This further analysis is shown in Figure 3b and described in lines 195-202. Critically, the further analysis exactly mirrors the selection criteria, i.e. it is aimed at testing the effect of Congruent/Incongruent and Upright/Inverted. This is colloquially known as "double dipping", the same contrast is used for selection (of channels, in this case) as for later statistical testing. This should be avoided, since in this case even random noise might result in a significant effect. To strengthen the evidence, either the authors could use a selection contrast that is orthogonal to the subsequent statistical test, or they could skip either the preselection step or the subsequent test. (It could be argued that the test in Figure 3b and related text is not needed to make the point - that same point is already made by the cluster-based permutation test.)
Related to the above: the test for the three-way interaction (lines 211-216) is reported as "marginally significant", with a p-value of 0.087. This is not very strong evidence.
-
Reviewer #2 (Public Review):
Summary:
The authors evaluate spectral changes in electroencephalography (EEG) data as a function of the congruency of audio and visual information associated with biological motion (BM) or non-biological motion. The results show supra-additive power gains in the neural response to gait dynamics, with trials in which audio and visual information were presented simultaneously producing higher average amplitude than the combined average power for auditory and visual conditions alone. Further analyses suggest that such supra-additivity is specific to BM and emerges from temporoparietal areas. The authors also find that the BM-specific supra-additivity is negatively correlated with autism traits.
Strengths:
The manuscript is well-written, with a concise and clear writing style. The visual presentation is largely …
Reviewer #2 (Public Review):
Summary:
The authors evaluate spectral changes in electroencephalography (EEG) data as a function of the congruency of audio and visual information associated with biological motion (BM) or non-biological motion. The results show supra-additive power gains in the neural response to gait dynamics, with trials in which audio and visual information were presented simultaneously producing higher average amplitude than the combined average power for auditory and visual conditions alone. Further analyses suggest that such supra-additivity is specific to BM and emerges from temporoparietal areas. The authors also find that the BM-specific supra-additivity is negatively correlated with autism traits.
Strengths:
The manuscript is well-written, with a concise and clear writing style. The visual presentation is largely clear. The study involves multiple experiments with different participant groups. Each experiment involves specific considered changes to the experimental paradigm that both replicate the previous experiment's finding yet extend it in a relevant manner.
Weaknesses:
The manuscript interprets the neural findings using mechanistic and cognitive claims that are not justified by the presented analyses and results.
First, entrainment and cortical tracking are both invoked in this manuscript, sometimes interchangeably so, but it is becoming the standard of the field to recognize their separate evidential requirements. Namely, step and gate cycles are striking perceptual or cognitive events that are expected to produce event-related potentials (ERPs). The regular presentation of these events in the paradigm will naturally evoke a series of ERPs that leave a trace in the power spectrum at stimulation rates even if no oscillations are at play. Thus, the findings should not be interpreted from an entrainment framework except if it is contextualized as speculation, or if additional analyses or experiments are carried out to support the assumption that oscillations are present. Even if oscillations are shown to be present, it is then a further question whether the oscillations are causally relevant toward the integration of biological motion and for the orchestration of cognitive processes.
Second, if only a cortical tracking account is adopted, it is not clear why the demonstration of supra-additivity in spectral amplitude is cognitively or behaviorally relevant. Namely, the fact that frequency-specific neural responses to the [audio & visual] condition are stronger than those to [audio] and [visual] combined does not mean this has implications for behavioral performance. While the correlation to autism traits could suggest some relation to behavior and is interesting in its own right, this correlation is a highly indirect way of assessing behavioral relevance. It would be helpful to test the relevance of supra-additive cortical tracking on a behavioral task directly related to the processing of biological motion to justify the claim that inputs are being integrated with the service of behavior. Under either framework, cortical tracking or entrainment, the causal relevance of neural findings toward cognition is lacking.
Overall, I believe this study finds neural correlates of biological motion, and it is possible that such neural correlates relate to behaviorally relevant neural mechanisms, but based on the current task and associated analyses this has not been shown.
-
Reviewer #3 (Public Review):
Summary:
The study demonstrates differential patterns of entrainment to biological motion (BM). At a basic, sensory level, the authors demonstrate entrainment to faster rhythms that make up BM (step-cycle) which seems to be separate from its audio aspects and its visual aspects (though to a much lesser degree). Ultimately this temporal scale seems to reside in a manner that does not indicate much multi-modal integration. At a higher-order, emergent rhythms in motion that are biologically relevant (gait-cycle) seem to be the result of multisensory integration. The work sheds light on the perceptual processes that are engaged in perceiving BM as well as the role of multisensory integration in these processes. Moreover, the work also outlines interesting links between shorter and longer integration windows …
Reviewer #3 (Public Review):
Summary:
The study demonstrates differential patterns of entrainment to biological motion (BM). At a basic, sensory level, the authors demonstrate entrainment to faster rhythms that make up BM (step-cycle) which seems to be separate from its audio aspects and its visual aspects (though to a much lesser degree). Ultimately this temporal scale seems to reside in a manner that does not indicate much multi-modal integration. At a higher-order, emergent rhythms in motion that are biologically relevant (gait-cycle) seem to be the result of multisensory integration. The work sheds light on the perceptual processes that are engaged in perceiving BM as well as the role of multisensory integration in these processes. Moreover, the work also outlines interesting links between shorter and longer integration windows along the sensory and multisensory processing stages.
In a series of experiments, the authors sought to investigate the role of multisensory integration in the processing of biological motion (BM). Specifically, they study neural entrainment in BM light-point walkers. Visual-only, auditory-only, and audio-visual (AV) displays were compared under different conditions.
Experiments 1a and b mainly characterized entrainment to these stimuli. Here, entrainment to step cycle (at different scales for 1a and 1b) was found to entrain in the presence of the auditory rhythm and to a certain degree also for the visual stimulus (though barely beyond the noise floor in 1b). The AV condition for this temporal scale seemed to follow an additive rule whereby the combined stimulation resulted in entrainment more or less equal to the sum of the unimodal effects. At the slower, gait cycle a slightly different pattern emerges whereby neither unimodal stimulation conditions result in entrainment however the AV condition does.
This finding was further explored in Experiment 2 where two extra manipulations were added. Point-light walkers could generally be either congruently paired with AV or incongruently. In addition, the visual BM stimulus was matched with a control consisting of an inverted BM and thus non-BM movement. This study enabled further discerning among the step- and gait-cycle findings seeing that the pattern that emerged suggested that step-cycle entrainment was consistent with a low-level process that is not selective to BM whilst gait-cycle entrainment was only found for BM. This generally replicated the findings in Experiment 1 and extended them further suggesting that entrainment seen for uni- and multisensory step cycles is reflects a different process than that captured in the gait-cycle multi-modal entrainment. The selective BM finding seemed to demonstrate a link to autistic traits within a sample of 24 participants informing a hypothesis that sensitivity to biological motion might be related to social cognition.
Strengths:
The main strengths of the paper relate to the conceptualization of BM and the way it is operationalized in the experimental design and analyses. The use of entrainment, and the tracking of different, nested aspects of BM result in seemingly clean data that demonstrate the basic pattern. The first experiments essentially provide the basic utility of the methodological innovation and the second experiment further hones in on the relevant interpretation of the findings by the inclusion of better control stimuli sets.
Another strength of the work is that it includes at a conceptual level two replications.
Weaknesses:
The statistical analysis is misleading and inadequate at times. The inclusion of the autism trait is not foreshadowed and adequately motivated and is likely underpowered. Finally, a broader discussion over other nested frequencies that might reside in the point-light walker stimuli would also be important to fully interpret the different peaks in the spectra.
-
-
-
