A capsule-based hierarchical graph reasoning model incorporating homologous and heterogeneous information for sentiment analysis

Although cross-modal complementary mechanisms dominate multimodal sentiment analysis, simultaneously acquiring multisource data such as text and audio in practical applications remains challenging. To address this issue, this paper proposes a capsule-based hierarchical graph reasoning model that incorporates homologous and heterogeneous information. This model jointly models the waveform and mel-frequency cepstral coefficients (MFCCs) of audio signals, leveraging their complementarity to enhance sentiment recognition capabilities under single-source data conditions. In the spectral coefficient mode, a region-level sparse reasoning module is introduced to combine local correlations within the spectrum for region-level frequency domain reasoning and feature extraction, thereby perceiving local sentiment features. Subsequently, an emotion capsule graph reasoning module is designed. This module employs a dynamic routing mechanism to map local features into higher-level capsules, capturing emotional representations across various segments of the spectrum. Additionally, it constructs semantic association graphs between higher-level capsules and spectral segments, mining emotional connections between segments to achieve sentiment modeling of entire sentences. In the audio mode, a multi-scale temporal feature reasoning fusion method is proposed. This method utilizes multi-scale convolutional networks to comprehensively extract local features from audio signals and bidirectional LSTM to capture contextual dependencies in the time dimension, thereby enhancing the ability to model emotional evolution. Finally, data from both modes are fused and used for category prediction. Experimental results demonstrate that the proposed method achieves competitive performance on several mainstream datasets.