Natural-language cohort queries on a hospital data platform: accuracy and provenance in a real-world evaluation

Read the full article See related articles

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

Background

Accessing routinely collected clinical data for research remains technically demanding across heterogeneous hospital systems. This proof-of-concept study aimed to evaluate the accuracy and query-level traceability of a commercial natural-language (NL) data-query platform implemented at a university hospital.

Methods

An on-premises data integration platform with an NL query interface (AIRE; Briya, Tel Aviv, Israel) was connected to the institutional health data platform across three tertiary-care hospitals under an approved pseudonymization and governance framework. Implementation comprised regulatory and governance approval, and IT deployment. Twenty-five NL prompts addressing cohort identification and characterization in an adult ICD-10 C71-coded brain-tumor cohort (n = 1,594) were evaluated against manually written reference SQL and checked for a corresponding executed query in the exported log. Responses were classified into five categories. Prompts with unanswered or incorrect responses were re-posed in a new thread.

Results

The platform was implemented over two years. Fifteen of 25 responses were exact (n = 11), concordant after definition alignment (n = 1), or incorrect but traceable from the logged SQL (n = 3); six were plausible but incorrect without a logged query, and four remained unanswered. No incorrect unlogged result recurred in a new thread. Across the initial and repeat evaluations, 22 of 23 logged responses were correct or traceable, whereas all eight unlogged responses were incorrect.

Conclusions

Implementation was feasible but required substantial local institutional preparation. The NL interface reproduced SQL-derived results when definitions were explicit, while query-level provenance and verification of the displayed result were critical for identifying unreliable outputs.

Article activity feed