Hierarchical Attention Medical Transformer for Enhanced Breast Cancer Classification from MRI

Breast cancer diagnosis from MRI is challenging, hampered by complex image interpretation and scarce annotated medical datasets, which limits deep learning efficacy. To address these challenges, we propose the Hierarchical Attention Medical Transformer (HAMT), a novel Vision Transformer for enhanced breast cancer MRI classification, especially in data-limited scenarios. HAMT integrates a medical domain-specific self-supervised pre-training strategy to learn relevant features and a hierarchical attention aggregation mechanism to synthesize information from multiple MRI slices into a robust patient-level representation. Evaluated on the Duke Breast Cancer Dataset, HAMT consistently outperformed state-of-the-art Vision Transformer baselines across various data scales. Notably, even with limited training data, HAMT demonstrated strong diagnostic performance, significantly surpassing the best baseline ensemble model. Ablation studies further confirmed the significant contributions of both proposed components. HAMT demonstrates superior generalization in data-scarce scenarios, offering a promising AI-powered tool to augment breast cancer diagnostic workflows.