Deep Learning Optimization of Automated Rectal Volume Segmentation in CT Imaging for Prostate and Cervical Cancers: Evaluation of Model Performance and Clinical Usefulness

Background: Precise delineation of the rectum is crucial in treatment planning for cancers in the pelvic region, such as prostate and cervical cancers. Manual segmentation is also still time-consuming and suffers from inter-observer variability. Additionally, deep learning has offered promising solutions, but their clinical utility requires comprehensive validation against real-world standards. Methods: In this study, a U-Net-based deep learning model was developed and validated for automated rectum segmentation on planning computed tomography (CT) scans from 200 patients with prostate or cervical cancer. Model performance was evaluated using spatial similarity metrics, including the Dice Similarity Coefficient (DSC), Hausdorff Distance, and Average Surface Distance. In addition, a radiation oncologist performed a prospective clinical evaluation using a 3-point Likert scale. To further assess the model’s ability to capture subtle anatomical variations, a novel sex prediction task was introduced. Statistical significance was examined using Wilcoxon signed-rank tests, Welch's t-tests, and Mann-Whitney U tests. Results: The U-Net–based model achieved high segmentation accuracy, with mean DSC values of 0.91 (IQR: 0.88–0.93) for prostate cancer and 0.89 (IQR: 0.86–0.92) for cervical cancer, showing no significant difference between the two groups (p = 0.12). The automated contours generated by the U-Net model covered 99.1% of the GTV, with minimal deviations (≤2 mm). During clinical evaluation, 89.2% of the contours were rated as excellent, and 9.1% required only minor adjustments. The model also demonstrated high performance in predicting patients’ biological sex from CT data (accuracy = 94.6%; AUC = 0.98). Compared to manual segmentation (12.7 minutes), the automated process reduced the average duration to 4.3 minutes. Conclusion: The deep learning model achieved highly accurate rectal segmentation and showed improved detection of soft tissue structures. This method supports incorporating artificial intelligence (AI)-based segmentation into radiotherapy workflows, boosting both the reliability and efficiency of treatment planning.