Automated HER2 scoring in breast cancer HE-stained images using hybrid deep learning ensemble and probabilistic optimization

Human epidermal growth factor receptor 2 (HER2) is a critical gene that serves as a receptor to transmit signals for aggressive cell division in cancer cells. Hence, testing of HER2 is important in treatment to indicate candidates for HER2-targeted therapy. However, in the current gold standard, i.e., the immunohistochemistry (IHC) test, the scoring is based on the pathologist’s analysis, which has an inter- and intra-observer variation chance due to variability in staining assessment. Automating HER2 scoring using hematoxyline eosin (HE) stained images can overcome these limitations, providing more accurate and consistent results, thereby reducing healthcare costs and enhancing patient outcomes. In this work, we have presented an automated framework for classifying HER2 scores of breast cancer using HE-stained images. The developed framework uses three fine-tuned deep learning models, namely GoogLeNet, ResNet-50, and Vision Transformer (ViT). It applies the proposed hybrid weighted individual voting ensemble (WIVE) to combine the confidence scores of all the constituent models. This approach comprises two independent techniques: the Model-Specific Weights Optimization (MSWO), customizing weights for individual models, and the Class-Specific Weights Optimization (CSWO), fine-tunes weights for specific classes using Probabilistic model-building genetic algorithms (PMBGAs). The proposed framework surpasses the existing methods in the literature. The CSWO approach achieves an accuracy of 99.42% and a precision of 99.54%, while the MSWO approach attains an accuracy of 99.07% and a precision of 99.21%. This study outlines an economically feasible and efficient prognostic model with the potential to provide clinically significant inputs. The use of this algorithm might offer a possibility for the replacement of IHC testing, minimizing the variability in HER2 scoring, as well as simplifying the diagnostic process.