Performance of Contemporary Large Language Models on the National Prosthodontic Resident Examination: A Multi-Year Comparative Study
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Purpose
To compare the performance of three large language models (LLMs)—ChatGPT-o1, Grok 3, and Gemini 2.0 Pro—on the American College of Prosthodontists’ National Prosthodontic Resident Examination (NPRE).
Materials and Methods
Official NPRE examinations from years 2015-2024 were retrieved from the American College of Prosthodontists website and used to prompt each LLM. Paid access tiers and advanced reasoning were enabled. Responses from LLMs were recorded and compared against the official answer keys for accuracy, variability across years, and consistency of responses. Duplicate questions were retained for the primary analysis and removed in a sensitivity analysis. Accuracy (percent correct) was compared using generalized estimating equations binary logistic regression (primary); Cochran’s Q with Bonferroni-adjusted McNemar tests (sensitivity). Year-to-year variability (SD of annual accuracy) was compared using weighted Levene tests. Consistency across duplicate items was quantified as the percentage of perfect matches and compared using Cochran’s Q with Bonferroni-adjusted McNemar tests.
Results
Across 1,487 questions, overall accuracy differed among models (global P < 0.001). ChatGPT-o1 achieved 78.4% correct, Grok 3 76.4%, and Gemini 2.0 Pro 73.2%; ChatGPT-o1 and Grok 3’s accuracy significantly exceeded Gemini 2.0 Pro (both P ≤ 0.016), with no significant difference between ChatGPT-o1 and Grok 3 (P = 0.099). After removing duplicates (n = 1,200), results were similar: ChatGPT-o1, Grok 3, and Gemini 2.0 Pro were 77.1%, 76.6%, and 71.5% correct, respectively; ChatGPT-o1 and Grok 3’s accuracy was significantly higher than Gemini 2.0 Pro (both P < 0.001). Variability across years did not differ significantly (P ≥ 0.097). Grok 3 showed significantly higher perfect-match consistency (79.2%) than ChatGPT-o1 (69.4%) and Gemini 2.0 Pro (64.5%) (P ≤ 0.010).
Conclusions
Contemporary LLMs showed 70–80% accuracy on postgraduate prosthodontic multiple-choice content. While promising for educational support, residual error and inconsistency warrant human oversight and transparent methods reporting before curricular adoption.
