Scoring the NIH Stroke Scale from bedside exam transcripts: a synthetic benchmark of a decision model and a large language model
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Clinicians score the NIH Stroke Scale (NIHSS) at the bedside, often under time pressure, and ambient AI tools that listen to clinical encounters are now entering practice. It is not known whether software can score the NIHSS from the exam conversation itself, rather than extract a score that has already been written in a note.
Methods
We created 112 synthetic code-stroke cases. One hundred were allocated in proportion to real code-stroke presentations, and 12 stress cases tested special scoring rules. For each case, a large language model (Claude Opus 5.5) wrote a reference score form (the answer key) and a speech-to-text-style exam transcript, with main-set transcripts split evenly between a narrated and a telegraphic examiner style. A second model (Claude Fable 5.1), blind to the key, scored every transcript and agreed with the key on 99.2% of items. Each transcript was then scored by Jev, a decision model, and by gpt-6-luna, a general large language model (LLM). The primary outcome was item-level accuracy against the key.
Results
The LLM scored 99.0% of items correctly (95% CI 98.4–99.4) and Jev 94.4% (93.2–95.4; McNemar p < 0.001), with exact totals in 101 and 63 of 112 cases. Jev was faster (median 0.15 vs 5.78 s per exam) and cheaper (USD 0.23 vs 0.67 per 1,000 exams). With telegraphic examiners, accuracy fell from 98.0% to 90.9% for Jev and from 100% to 97.7% for the LLM. In a post-hoc analysis that kept only the examiner’s words, accuracy stayed high when the examiner narrated (Jev 96.9%, LLM 98.8%) but dropped to about 80% when the examiner used shorthand. Most of Jev’s errors fell on four items (language, extinction and neglect, LOC questions and visual fields), and on each it tended to repeat the same mistake, for example scoring slurred speech as aphasia. Its confidence flagged most of these errors. Re-checking the 14.8% of items with confidence below 0.70 would have caught 86% of them, and a simulated hybrid reached 98.8% accuracy.
Conclusions
Both engines scored most NIHSS items correctly from exam transcripts. The LLM was more accurate, and the decision model was faster and cheaper, with a confidence signal that could route uncertain items to an LLM. Because the cases and the key are synthetic, these findings need confirmation with recorded exams and human raters.
