Research on Multimodal Hate Speech Detection Based on Self-Attention Mechanism Feature Fusion

The widespread rise of multimedia social platforms has diversified the ways in which people communicate and the content they share. Hate speech, as a threat to societal harmony, has also shifted its manifestation from a singular textual to a multimodal one. Previously, most methods for detecting hate speech were limited to the text modality, making it difficult to identify and classify newly emerging multimodal hate speech that combines text and images. This paper proposes a novel multi-modal hate speech detection model to respond to the above-mentioned needs for multi-modal hate speech detection. The proposed joint model can use moving windows to extract multi-level visual features and extract text features based on the RoBERTa pre-training model and introduces a multi-head self-attention mechanism in the later fusion process for image and text feature fusion. This article also conducted experiments on the multi-modal benchmark data set Hateful Memes. The model achieved an accuracy of 0.8780, precision of 0.9135, F1-Score of 0.8237, and AUCROC of 0.8532, defeating the SOTA multi-modal hate speech recognition model.