COSMOS-CVNet: A Cross-modal OCR-Spam Model with Optical Stream and CNN-ViT Network for Enhanced Image Spam Detection

Recently, spam image detection has turned out to be an increasingly intricate problem due to the evolving tactics employed by spammers, and embedding malicious content within images in sophisticated ways. Several Artificial Intelligence (AI)-aided algorithms were developed priorly intending to detect spam. However, the evolving nature of spam makes it complex for Machine Learning (ML) or Deep Learning (DL) algorithms to effectively identify them. For this reason, this paper proposes a novel hybrid model named COSMOS-CVNet, a Cross-modal OCR-Spam Model with Optical Stream and CNN-ViT Network. The proposed COSMOS-CVNet is developed by combining Convolutional Neural Network (CNN)-based image enhancement, Optical Character Recognition (OCR)-based text feature extraction, and Cross-Modal Attention (CMA) to detect spam together with enhanced CNN and Vision Transformer (ViT) networks. The model begins with preprocessing which includes CNN-based image enhancement, contrast improvement using Contrast Limited Adaptive Histogram Equalization (CLAHE), and bilateral filtering for noise reduction. The enhanced images are now passed through a pre-trained ResNet50 model to extract discriminative visual features. Simultaneously, Tesseract OCR is used to extract any embedded textual information from the images. A CMA mechanism is now used to fuse these features based on the weights of the visual and textual features to enhance model performance. Finally, the fused features are classified using a hybrid classification layer that integrates a ViT for spatial attention learning and a CNN for fine-grained feature extraction. The model is developed on a labeled dataset of spam and non-spam images, and the results of experiments demonstrate that the hybrid approach is effective in reaching superior accuracy (99%) in detecting evolving and novel spam patterns. Besides, a comparison study is carried out to evaluate the effectiveness of the proposed COSMOS-CVNet in detecting image spams over SOTA models.