Human In the Loop Challenges for Quality Annotation of Pre-Cancer Lesions in Clinical Oral Images

Smartphone-based Artificial Intelligence (AI) enabled screening of the hyperplasia and dysplasia stages (pre-cancer, OPMD) offers a viable opportunity to reduce the incidence and mortality of oral cancer through early prevention. However, developing accurate segmentation models requires high-quality, pixel-level annotations that are prohibitively expensive and prone to clinical subjectivity. To address this, we empirically validated a deep learning-driven Human-in-the-Loop (HITL) iterative pseudo-labeling framework to pixel-annotate 3,026 clinical oral images. We also conducted controlled experiments to quantify the network’s tolerance to label noise (unreviewed pseudo-labels) and resolved clinical subjectivity using pixel-wise Cohen’s Kappa and the STAPLE consensus algorithm. While iterative self-training consistently improved lesion detection and spatial localization, including even a modest fraction ( → 10%) of unreviewed pseudo-labels increased training convergence instability three-to-four-fold and induced a conservative prediction bias that negatively impacted model recall. Ultimately, the model’s performance converged with the inter-rater reliability ceiling ( κ → 0.65) against a multi-expert ground truth, successfully falling within the envelope of human agreement. These findings highlight that non-experts can successfully drive the early-to-mid stages of the annotation pipeline, but a final expert-driven quality assurance step is strictly essential to mitigate training instability, confirmation bias, and clinically unacceptable recall drops. Overall, this provides a scalable, empirically validated blueprint for building domain-specific medical imaging datasets in global health settings where annotation cost, inter-observer variability, and expert unavailability challenges are most acute.