Leveraging Large Language Models to Direct Automated PET/CT Tumour Segmentation in Retrospective Data: an Agentic Framework Method

Automated medical image segmentation using deep learning requires large labelled datasets, presenting barriers for rare cancers like sarcoma. We developed an agentic framework integrating LLM analysis of radiologist reports with nnUNet segmentation for ¹⁸ F-FDG PET/CT imaging data (N=60, 134 studies), aiming to improve automated tumour segmentation for retrospective data. LLM interpretation was optimised in an expanded report dataset (N=91, 226 studies). The framework aimed to first screen for disease presence, then tested localising tumours to body regions mapped via automated organ segmentation, enabling targeted image cropping before segmentation. A baseline nnUNet was trained for comparison and achieved a mean Dice of 0.49. The disease classification framework attained mean Dice 0.62, correctly identifying 9/11 disease-free cases versus 6/11 for baseline. This approach demonstrates potential for leveraging routinely collected clinical text data to enhance medical imaging research in rare diseases.