Evaluating the Interplay and Impact of Quality and Technical Variations on Performance of AI Models in Multi-Institutional Prostate MRI Datasets

Purpose

To evaluate whether visually assessed high-quality prostate MRI, as defined by the Prostate Imaging Quality (PI-QUAL) scoring system, is sufficient to ensure consistent AI model performance, or whether hidden acquisition-related technical effects introduce variability that cannot be detected through visual inspection.

Methods

This retrospective study included 568 bpMRI scans from seven institutions, classified using PI-QUAL v2 into high-quality (scores 2-3) and low-quality (score 1) groups. Quantitative image quality features were extracted using MRQy from T2w and ADC sequences. Three experiments were conducted on high-quality scans: (1) unsupervised clustering to identify hidden technical batch effects; (2) V-Net models (M₁) for prostate capsule segmentation on T2w using single-cluster and balanced multi-cluster training strategies; and (3) VoxelMorph-based deformable registration (M₂) to align ADC maps to T2w images. Performance metrics included DSC, precision, recall, volume ratio, HD95, ASSD, and TRE. Kruskal-Wallis tests assessed cross-cluster variability, Two One-Sided Tests (TOST) evaluated performance equivalence, and bootstrap resampling quantified training efficiency.

Results

Clustering identified four clusters among 277 high-quality bpMRI scans. Single-cluster models showed variable performance (DSC=0.41-0.85) with significant batch effects (p<0.001), even with 140 training samples. The balanced multi-cluster model (n=60) eliminated batch effects (p=0.075), achieved equivalent performance (p<0.001), and demonstrated 2.23-fold greater training efficiency despite using 2.3-fold fewer samples. Registration performance also varied significantly across clusters (TRE=0.86-5.47 mm, p<0.001).

Conclusion

Acquisition-related batch effects significantly impact AI generalizability in prostate MRI, even among high-quality scans. Quantitative clustering enables detection of technical variations and supports diversity-aware training strategies that achieve equivalent performance with 2.3-fold fewer samples while eliminating batch effects.

Summary

Acquisition-related batch effects significantly impair AI generalizability in prostate MRI, even among scans meeting PI-QUAL quality standards. Clustering-based diversity-aware training strategies successfully eliminated these batch effects and achieved 2.23-fold greater training efficiency compared to single-site approaches, demonstrating that strategic technical diversity outweighs dataset size for developing robust, clinically deployable AI models.

Key Points

• Acquisition-related batch effects significantly impair AI performance on prostate MRI, exceeding the impact of visual image quality as assessed by PI-QUAL scoring.

• Even the model trained on the largest single cluster exhibited significant batch effects (p<0.001). In contrast, a balanced multi-cluster model eliminated cross-cluster variability (p=0.075) while maintaining equivalent performance and requiring a smaller sample size.

• Diversity-aware training achieved 2.23-fold greater training efficiency (95% CI: 2.18-2.29) with 2.3-fold fewer samples, demonstrating that technical diversity outweighs data quantity for robust AI generalization.

Clinical relevance

Addressing batch effects is essential for developing reliable, generalizable AI tools for prostate MRI interpretation.