MaxCAT-Net: Hybrid transformer-based skin lesion classification and lightweight segmentation with calibration and cross-dataset validation

Class imbalance, poor calibration of confidence and poor generalisation under different imaging conditions are restrictions of automated analysis of dermoscopic skin lesions. We propose a hybrid classifier MaxCAT-Net (MaxViT + CBAM + Attention Transformer), which consists of a pre-trained MaxViT-Tiny backbone and the combination of CBAM and two-layer lightweight Transformer encoder and evaluate four encoder-decoder networks for lesion segmentation as a separate task on ISIC 2018 Task 1. All the images of exact/near duplicate were removed and the remaining 10,012 images were distributed lesion-wise (72/14/14) without two or more images having the same type of lesion in HAM10000. On the test set ( n = 1,405), MaxCAT-Net outperformed the baselines of ResNet152-CBAM, MobileNetV3-CBAM, GhostNet-hybrid and single MaxViT-Tiny-CBAM with an accuracy of 0.869 (95% CI 0.847–0.890), balanced accuracy of 0.742, macro-F1 of 0.751 and macro-AUC of 0.944, although ablation gains in macro-F1 (0.017–0.022) were not statistically conclusive. By scaling the temperature, the ECE was decreased from 0.099 to 0.038. In the case of the PAD-UFES-20 dataset, however, the accuracy fell to 0.341 and the melanoma recall rate to 0.096 while for the HAM10000 dataset, the melanoma recall rate was only 0.587. A Swin-T-encoder U-Net was able to achieve a Dice score of 0.903 in segmentation (520 test images), whereas a FasterNet-T0 U-Net (1/10th of the number of parameters) had a Dice score of 0.901 (Holm adjusted p < 0.002), and a standard U-Net had a significantly lower Dice score of 0.888. All results are based on one seed and show trends—not definite improvement.