Predicting metastatic potential of primary cutaneous melanomas utilizing weakly supervised vision language model

Cutaneous melanoma is an aggressive form of skin cancer. Knowledge if a primary melanoma is likely to metastasize is crucial for treatment and survival prediction of melanoma patients. We aimed to develop a predictive tool for determining metastatic potential in primary melanomas utilizing a weakly supervised vision language model. A total of 426 routine stained whole slide images (WSI), along with corresponding histopathological features (Breslow thickness, diameter, presence of dermal mitoses, ulceration and regression), were collected. Of these, 341 samples were used for training and validation, while 85 were reserved as a holdout test set. WSIs were split into patches, and feature embeddings were extracted using Prov-GigaPath. Histopathological features were converted to text, with embeddings generated by BiomedBERT. We developed a multimodal transformer integrating WSIs and histopathological features and conducted an ablation study comparing it to (1) TransMIL using only WSIs and (2) an MLP using only histopathological features. Each model employed a bagging ensemble with five cross-validation models. The multimodal transformer achieved an AUC of 0.887, slightly higher than TransMIL (0.883) and notably better than BertMLP (0.800), highlighting the benefit of including imaging and clinical data for early recognition of melanomas with high metastatic potential.