From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objectives
Patients and clinicians increasingly ask large language models (LLMs) about Kampo, Japan’s traditional medicine. Yet evaluations of LLMs have measured accuracy on examination questions, which ask what a model knows; a consultation asks whether it can apply that knowledge to the patient at hand. No benchmark had measured that.
Materials and Methods
KampoBench comprises 74 consultation scenarios and 310 conversation-specific rubric criteria, following the design of HealthBench. Scenarios, criteria and reference answers were drafted by an LLM and revised by board-certified Kampo physicians. Ten current models from three vendors were evaluated. Because LLM graders are known to favor their own family, each response was graded by a model from each vendor.
Results
The best model reached 74.8-86.3% across graders, the physician-revised reference answers 87.9-92.4%. Rankings were stable across graders (Spearman’s rho 0.952-0.976), and the family-favoring deviation was 0.4 to 0.6 percentage points, below the bootstrap standard deviation of one measurement; grader leniency varied by 14.3 points. Models were most reliable on criteria about advising that care be sought (83.6%) and least reliable on those requiring a known crude-drug risk to be tied to the case (17.5%), though 79.2% of those responses named the drug.
Discussion
The shortfall lay in applying knowledge to the case rather than in holding it — a distinction examination-format evaluations cannot make.
Conclusion
Evaluation in traditional medicine should measure the judgments a consultation requires, not recall alone. Quoted alongside the grader that produced it, a KampoBench score is a usable measure of those judgments.
Lay summary
People increasingly ask artificial intelligence chatbots about Kampo, the traditional herbal medicine used alongside conventional care in Japan. Earlier tests used examination questions, which check what a chatbot knows. A real consultation asks something harder: whether the chatbot can apply that knowledge to the person asking. We built KampoBench, 74 realistic consultations in Japanese. For each one, specialist Kampo physicians reviewed a list of what a good answer should and should not contain, 310 items in all. We then tested ten current chatbots from three companies. The answers were scored by chatbots, and a chatbot might favor its own company’s chatbots, so every answer was scored by one chatbot from each company. No such favoritism was detected, but the scorers differed in strictness. The best chatbot scored 75-86%, depending on the scorer; answers revised by physicians scored 88-92%. The chatbots were good at telling people to seek medical care. They were poor at relating a known risk of an herbal ingredient to the person’s situation, such as noticing that two medicines the person already takes contain the same ingredient, often even when they named it. Tests of chatbots in traditional medicine should measure such judgments, not knowledge alone.
