A Vision-Language Framework for Predicting Brain Tumor Recurrence from Multimodal, Longitudinal Patient Data

Accurate prediction of tumor recurrence in brain tumor patients following surgery is essential for optimizing adjuvant therapy, response assessment, and surveillance regimen. While MRI remains the gold standard for surveillance, integrating patient-specific clinical context may inform recurrence prediction. Traditional multimodal deep learning approaches often incorporate clinical data via simple fusion, failing to fully capture the semantic interdependencies between visual features and clinical context. Trained on over 5,000 scans from approximately 400 pediatric low-grade glioma subjects and validated across three institutional cohorts, including one clinical trial cohort, our experiments demonstrate incremental performance gains when progressing from vision-only to clinical-vision to a vision-language approach. Our results indicate that converting structured clinical covariates into natural language text allows for more effective synthesis of multimodal data, while providing a platform for incremental addition of clinical context without extending model complexity. We demonstrate that our proposed VLM architecture offers a promising direction for neuro-oncological prognosis by effectively encoding imaging cues and clinical context, with potential applicability to other longitudinal prognosis tasks.