Hybrid Metaheuristic Feature Selection for Enhanced Breast Cancer Detection in Digital Mammography: A Radiomics and Deep Learning Approach with Cross-Dataset Validation

Background Artificial intelligence (AI) shows promise for improving breast cancer detection in mammography, but generalizability across datasets and imaging conditions remains a major challenge. We developed a hybrid metaheuristic feature-selection framework that combines radiomics and deep learning features and evaluated it on a real pilot dataset and a controlled synthetic comparison. Methods A diagnostic model was developed using the public CBIS-DDSM dataset. The framework combined 2,051 IBSI-compliant radiomic features and 2,048-dimensional deep features from a pretrained EfficientNet-B5 model. A hybrid Grasshopper Optimization Algorithm and Crow Search Algorithm (GOA-CSA) was used to select an optimal feature subset for an MLP classifier. A controlled synthetic comparison (N = 16, D = 1114) compared an inventive multi-constraint fitness function against a legacy fitness under collapse-prone conditions. Results On a CBIS-DDSM pilot subset (n = 22, 5-fold cross-validation), the hybrid GOA-CSA model achieved an AUC of 0.858 while reducing the feature count by 95% to 102 features, compared with an all-features baseline AUC of 0.825. In the synthetic comparison, the inventive fitness achieved AUC 0.810 and sensitivity 0.571 versus 0.476 and 0.286 for the legacy fitness. The collapse-prevention mechanism was implemented but was not triggered in this synthetic run, as both models maintained sensitivity greater than zero. Conclusions The hybrid metaheuristic framework improved feature selection performance on both the real pilot and synthetic comparison. The synthetic experiment supports the value of the multi-constraint fitness design, but real-data validation of collapse prevention remains necessary.