SAFARI: Self-supervised and Adversarial Feature Analysis for Robust Image retrieval

Software maintenance is a crucial phase for software enterprises after product delivery. In practice, users often provide screenshots of errors as evidence when reporting issues, and maintenance engineers need to quickly locate software faults using these interface screenshots. To achieve rapid response in maintenance, retrieving images from a database of software interface screenshots becomes essential. To address this, we construct an image interface database and retrieve images using user screenshots to quickly identify the software and the malfunctioning features. This paper focuses on the task of image-based software interface retrieval and proposes an innovative image retrieval method that integrates adversarial learning and self-supervised networks. The algorithm combines a visual content branch (extracting global features) with a visual focus branch (extracting local features). The visual content branch employs a Vision Transformer (ViT) model, which divides the image into multiple patches and processes them through a Transformer encoder, effectively extracting global features. The visual focus branch utilizes a lightweight Dynamic Convolutional Neural Network (DCNN) to capture local image information. Additionally, an attention-based feature fusion module is designed to integrate global and local information. Experiments conducted on multiple public datasets validate the effectiveness of the proposed method in image retrieval tasks. Ablation studies further demonstrate the efficacy of the proposed innovations. When processing images containing textual information, the experimental results show that our method can effectively extract global semantic information from images, significantly improving image retrieval performance.