Uncertainty-aware extraction of clinical findings from Finnish EHRs using open large language models
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
To evaluate whether open-weight large language models (LLMs) can accurately extract clinical findings from Finnish-language pediatric records, and whether prediction uncertainty can be used to triage cases for expert review to minimize manual work
Materials and Methods
Retrospective cohort of 97 pediatric ischaemic stroke patients (1 month–17 years) from Helsinki University Hospital (2010–2023). Three open LLMs (gpt-oss-20b, DeepSeek-R1-Distill-Qwen-32B, and medgemma-27b-text-it) were prompted in English to detect four extraction targets (hemiplegia, headache, seizure, and stroke as a positive control) from each patient’s full free-text record. Each combination received 15 calls (five temperatures × three repeats). Performance was benchmarked against a clinician reference (accuracy, recall, precision, F1). Shannon entropy across the 15 calls quantified within-model uncertainty; inter-model disagreement provided an ensemble signal. Patients were ranked by uncertainty for a simulated selective-review workflow. Findings were externally validated in an independent neonatal stroke cohort (n = 88).
Results
Gpt-oss-20b achieved the best balance of recall (0.91–1.00) and precision (0.83–0.92), with F1 0.89–0.95 across non-control extraction targets. Entropy in misclassified cases was 2.4–3.4 times higher than in correctly classified cases. Entropy-based triage achieved complete error coverage by reviewing <10% of patients for hemiplegia (8.3%) and headache (8.2%), and 19.6% for seizure. Neonatal validation reached F1 0.95 for Apgar 1 min and binary seizure, and F1 0.87 for 4-class stroke-subtype classification.
Discussion
Within-model entropy and inter-model disagreement provided complementary, calibrated signals of likely error in a non-English clinical setting.
Conclusion
Open LLMs can extract clinical findings from Finnish pediatric records with accuracy comparable to published English benchmarks, and uncertainty-based triage substantially reduces required expert workload.
