Structured extraction from Russian/Kazakh discharge summaries: three model families at the human-agreement ceiling
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
To determine which model family a hospital should use to extract structured data (sections, medication records, laboratory tuples) from Russian/Kazakh stroke discharge summaries, bounded against measured human agreement.
Materials and Methods
Three families were compared under identical inputs: a multilingual BERT-style encoder with CRF heads, a locally fine-tuned Qwen3-4B, and zero-shot GPT-5.5, on three tasks of increasing granularity: section segmentation (4,327 annotated documents), seven-field medication records (708 documents), and dated lab tuples (1,888 documents). All arms shared a block-ID representation (extraction as tagging over a fixed segmentation). Blind re-annotation of 50 documents per task provided the human-agreement ceiling; model-vs-ceiling claims use paired same-document tests with Holm correction.
Results
On section segmentation all arms fall within ±0.03 of the ceiling (0.905; models 0.919-0.932). On medication detection the cloud model is above it (0.966 vs 0.959); on full-record assembly, re-scored symmetrically against a blind annotator as common reference, both the fine-tuned LLM and the cloud model reach 0.344 agreement versus the gold annotator’s 0.183. On lab assembly the fine-tuned LLM matches the full-tuple ceiling (0.898 vs 0.902) while the encoder is far below (0.476). Residual errors concentrate in annotation conventions the annotators themselves could not agree on.
Discussion
Detection is solved at the human ceiling even for non-English clinical text, and a minutes-scale fine-tune reaches it with no data egress. Assembly separates the families and is convention-bound: benchmark consumers should read “above human agreement” against what the guidelines fixed.
Conclusion
Annotation guidelines and task representation, not model capacity, are the binding constraints for this task family.
