Enhanced Cellular Detection in Cervical Cytopathology: A Systematic Study of YOLO11 Training Paradigms

Automated cellular detection using deep learning is a key strategy for optimising cervical cancer screening by reducing the healthcare workload and inter-observer variability. However, analysing Whole Slide Image (WSI) patches presents challenges such as annotation scarcity, morphological complexity, and class imbalance. This study systematically evaluates YOLO11-n, YOLO11-s, and YOLO11-m to assess the impact of target variable granularity and training paradigms on performance. Four strategies were analysed: independent and multi-class models, each evaluated at both the specific cell label and diagnostic macro-group levels. To ensure clinical robustness, patient-level data partitioning was implemented to prevent data leakage. Performance was measured using precision, recall, and mAP (0.5 and 0.5:0.95). The results reveal critical trade-offs between fine-grained discrimination and model generalisation when varying the architectural complexity and labelling strategies. The findings indicate that diagnostic aggregation improves stability, whereas single-class training optimises specialised detection. These results provide methodological guidelines for designing AI-assisted screening systems and may inform future extensions of WSI-level diagnostic pipelines.