Exploring Architectural Enhancements for HoVer-Net: Deep Learning-based Method of Automated Nuclear Segmentation and Classification

Accurate nuclei segmentation and classification are vital for computational pathology. This study builds upon the HoVer-Net architecture by integrating modern architectural components to enhance multi-task performance on the PanNuke dataset, which contains both segmentation and classification labels across 19 tissue types. We evaluate the effects of Squeeze-and-Excitation (SE) blocks, multi-head attention, enhanced DenseBlock decoders, and transformer-based encoders (ViT, SwinViT). All models follow HoVer-Net’s preprocessing, training, and loss functions for consistent comparison. Results show that adding SE blocks to the encoder improves overall performance by approximately 3.6% in Dice scores, while transformer-based encoders lead to slight performance degradation. Our best model, MSDHV-Net (Multi-head Attention + SE + enhanced decoder), consistently outperforms the original HoVer-Net across several nuclei classes without increasing computational cost. These findings highlight the value of targeted architectural enhancements in advancing nuclei analysis models.