Patient-level consequences of local updates to retinal AI for diabetic retinopathy screening: a retrospective two-cohort evaluation

Objectives

To assess whether local artificial intelligence updates for diabetic retinopathy screening that improve image-level probabilities also improve patient-level detection, flagging and net benefit.

Methods

This retrospective evaluation used four systems, BRSET source data and two external cohorts, mBRSET and DeepDRiD, with separate audit and evaluation patients. Updates were fitted and selected on audits of 25–200 patients; probability recalibration and operating policies were selected separately. Outcomes included Brier loss, patient detection, flags and net benefit at four threshold probabilities (p=0.05–0.30). Comparators were no update, flag all and flag none. Conditional patient-cluster intervals compared updates with no update; simple-policy comparisons were descriptive.

Results

At audit budget 200, mean Brier loss improved in all eight cohort–system comparisons. In mBRSET, flags rose from 154–442 to 809–951 per 1,000 patients and missed positives fell from 51–133 to 2–14 (5.69–9.48 additional flags per additional detection). In mBRSET at p=0.05, three of four updates had lower mean net benefit than flag all; at p=0.20, all four exceeded flag all but remained below no update. In DeepDRiD at p=0.20, the APTOS update increased mean net benefit by 0.0510 over no update and exceeded both simple policies.

Discussion

Image-level improvement did not ensure patient-level benefit. Flag increases were consistent with image-level targets applied to patients with about four images.

Conclusion

Local updates should be judged on patient-level detection, flags and net benefit against unchanged, flag-all and flag-none policies, not on image-level metrics alone.

KEY MESSAGES