Board-Level Performance of Leading Open-Weight Vision-Language Models on the Japanese Diagnostic Radiology Board Examination: Reasoning, Image-Input, and Language Effects

Purpose

To evaluate the latest open-weight vision-language models (VLMs) on the Japanese Diagnostic Radiology Board Examination (JDRBE), assessing overall accuracy and the effects of image input, reasoning, and language.

Materials and Methods

In this retrospective study, 29 open-weight VLMs from 13 developers, released in or after January 2025, were evaluated on 327 image-bearing questions from four years of the JDRBE, a non-public benchmark with low risk of data leakage. Each question was answered by each model with and without the image(s), under three language conditions and with reasoning enabled and disabled. Accuracy was the primary outcome, and within-model differences were tested with paired bootstrap confidence intervals and sign-flip permutation tests with Benjamini–Hochberg correction.

Results

In the Japanese condition with image input and reasoning, the leading models reached 73.7% (gemma-4-31B-it), 73.1% (Qwen3.5-397B-A17B), and 72.1% (Kimi-K2.6). On the 2025 sub-set, these three models (74.1%–75.5%) scored above the mean accuracy of five newly board-certified radiologists who passed the 2025 examination (72%; range, 65%–83%). Accuracy broadly scaled with model size, although compact gemma-4-31B-it matched larger models. Enabling reasoning improved accuracy in nearly all models and the contribution of image input was larger when reasoning was enabled, particularly in higher-performing models. English prompts generally outperformed Japanese prompts.

Conclusion

Several open-weight VLMs, without medical adaptation, performed at or above the mean of newly board-certified radiologists on the JDRBE, with both model size and reasoning contributing. The highest Japanese-language accuracy came from a compact model suitable for parameter-efficient fine-tuning and serving on a single graphics processing unit.

Summary Statement

Several open-weight vision-language models, run on local infrastructure and without medical adaptation, performed at or above the mean of newly board-certified radiologists on the Japanese Diagnostic Radiology Board Examination.

Key Points

• In this retrospective study of 29 open-weight vision-language models evaluated on 327 image-bearing questions from the Japanese Diagnostic Radiology Board Examination, the leading models reached 72.1%–73.7% accuracy in the Japanese condition; on the 2025 subset, three models (74.1%–75.5%) scored above the mean of five newly board-certified radiologists who passed that examination (72%).

• Enabling reasoning improved accuracy in nearly all models across language conditions (mean within-model gain, +4.5 to +8.0 percentage points), and significant contributions of image input were more frequent in the reasoning-enabled condition (22 of 56 instances vs 11 of 61 with reasoning disabled).

• The highest Japanese-language accuracy was achieved by gemma-4-31B-it (73.7%), a compact model that can be served on a single graphics processing unit and fine-tuned with parameter-efficient methods; it also used the fewest tokens among the leading models.