Object Detection in Laparoscopic Surgery: A Comparative Study of Deep Learning Models on a Custom Endometriosis Dataset

Laparoscopic surgery for endometriosis presents unique challenges due to the complexity and variability of lesion appearances within the abdominal cavity. This study investigates the application of deep learning models for object detection in laparoscopic videos, aiming to assist surgeons in accurately identifying and localising endometriosis lesions and related anatomical structures. A custom dataset was curated, comprising 332 video sequences and 354,906 frames. Of these, 17,560 frames were meticulously annotated by medical professionals. The dataset includes object-detection annotations for 10 object classes relevant to endometriosis, alongside segmentation masks for some classes. To address the object detection task, we evaluated the performance of three deep learning models - FasterRCNN, MaskRCNN, and YOLOv9 - under both stratified and non-stratified training scenarios. Experimental results demonstrated that stratified training significantly reduced the risk of data leakage and improved model generalization. The best-performing object detection model achieved high precision and recall across most classes. Despite these successes, the study also highlights the challenges posed by the weak annotations and class imbalances in the dataset, which impacted overall models performances. In conclusion, this study provides valuable insights into the application of deep learning for enhancing laparoscopic surgical precision in endometriosis treatment. The findings underscore the importance of robust dataset curation and advanced training strategies in developing reliable AI-assisted tools for surgical interventions. Future work will focus on refining the dataset and exploring more sophisticated model architectures to further improve detection accuracy.