Pre-treatment multimodal artificial intelligence for prognostic stratification in locally advanced rectal cancer

Accurate prognostic assessment before neoadjuvant chemoradiotherapy remains challenging for locally advanced rectal cancer (LARC), limiting personalised treatment decisions. Here, we develop the Integrated Multimodal Prognostic Assessment for Locally Advanced Rectal Cancer Neoadjuvant Chemoradiotherapy (IMPACT), an artificial intelligence framework employing bidirectional multimodal attention mechanisms to capture cross-modal feature interactions and integrating pre-treatment pelvic magnetic resonance imaging, pathological biopsy whole slide images, and clinical information from 752 LARC patients across two independent centres. IMPACT achieves C-indexes of 0.805 for overall survival and 0.760 for disease-free survival, significantly outperforming the Guideline-based Imaging Risk Score (0.712 and 0.697, respectively). High-risk patients demonstrate 8.3-fold increased mortality risk and 6.5-fold increased recurrence risk compared to low-risk patients. External validation maintains robust performance with preserved risk stratification capability. Systematic ablation studies confirm the incremental value of trimodal fusion over single-modality approaches. IMPACT enables accurate pre-treatment prognostic stratification, facilitating evidence-based treatment intensification for high-risk patients and de-escalation strategies for low-risk cases in clinical practice.