Identifying interstitial lung disease with large language models on clinical notes and claims
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Interstitial lung disease (ILD) is a potentially fatal toxicity of immune check-point inhibitors and antibody-drug conjugates. Drug-safety studies in healthcare databases identify ILD with claims-based algorithms that trade positive predictive value (PPV) against sensitivity. We assessed whether large language models (LLMs) improve on these algorithms, with and without electronic health record (EHR) notes, and whether these gains translate into measurable epidemiological benefits.
Methods
In 309 oncology patients with ILD diagnosis codes at Mass General Brigham, adjudicated by two pulmonologists, we compared 12 open-access LLMs with 20 published claims-based algorithms. LLMs received claims, EHR notes, or both, without task-specific training. Measures were PPV and yield (proportion classified positive). We projected the attenuation of a drug-induced ILD odds ratio under non-differential outcome misclassification.
Findings
LLMs extended the PPV–yield frontier of the published algorithms with all three inputs. Across the 36 model–input evaluations, LLM PPV exceeded that of the best algorithm without specialist-coding requirements (Algorithm 4) by 14.3 percentage points on average. With claims alone, Qwen3-8B achieved 89.8% PPV (95% CI 78.2–95.6) at 15.9% yield, versus 86.7% at 4.9% for the highest-PPV algorithm. With notes alone, Qwen3-32B achieved 78.9% PPV at 49.2% yield, versus 64.6% at 51.1% for Algorithm 4. With combined claims and notes, Qwen3-14B achieved 100.0% PPV at 24.3% yield. Under a simulated drug-safety scenario with a true odds ratio of 3.0, the projected observed odds ratio was 2.67 with combined inputs versus 1.07 with Algorithm 4, corresponding to a 46-fold difference in the estimated sample size required for 80% power (591 versus 27,200 patients).
Interpretation
Zero-shot LLMs offer a higher-PPV alternative to rule-based claims pheno-typing for ILD, on all three input types. Because the claims-only configuration uses the same claims data as existing algorithms, it could be deployed in administrative databases without linked notes, and may recover drug-safety signal otherwise lost to outcome misclassification.
Funding
National Institute on Aging, National Library of Medicine, PCORI, and the National Institute of Arthritis and Musculoskeletal and Skin Diseases.
Research in context
Evidence before this study
We searched PubMed up to August 2026 for studies on claims or electronic health record (EHR)-based identification of interstitial lung disease and on machine-learning or large language model (LLM) phenotyping of clinical conditions. Search terms included “interstitial lung disease or pneumonitis”, “administrative claims”, “computational phenotyping”, “rule-based algorithms”, “machine learning”, and “large language models”. We also screened the reference lists of relevant articles. The literature search suggested that identifying ILD in longitudinal claims data is unreliable because the relevant diagnosis codes are nonspecific, and validated rule-based algorithms face a structural tradeoff between PPV and coverage. Supervised machine-learning classifiers have outperformed rule-based ILD identification on electronic health record data, and zero-shot LLMs have already shown the ability to phenotype chronic conditions from structured codes. No study, however, had compared a zero-shot LLM head-to-head against validated, published claims-based algorithms for a specific, high-stakes drug-safety outcome on a single fixed cohort, isolated the contribution of the input substrate from that of the model, or quantified the downstream consequence for pharmacoepidemiologic effect estimation.
Added value of this study
On a single, fixed, expert-adjudicated cohort, the evaluated LLMs achieved higher positive predictive value (PPV) than all 20 published claims-based algorithms at comparable yield (the proportion of patients classified as ILD-positive), whether the models received structured claims, clinical notes, or both. Across the 36 model–input evaluations, PPV exceeded that of the most inclusive published rule by 19.7 percentage points on average, and did so in 33 of 36 evaluations. We show that clinical notes generally increased yield, while the resulting PPV–yield trade-off depended on the model and input combination. Restricting the input to a single EHR note type preserved the advantage over rule-based phenotyping in matched comparisons on the patients possessing that EHR note type. Most consequentially, this improvement reduces the sample size required to detect a hypothesised drug-ILD association from approximately 27,200 patients for the best existing algorithm for the task to 591 when using LLMs, and offer a deployable upgrade for the note-free administrative databases where rule-based algorithms are the current standard.
Implications of all the available evidence
For pharmacovigilance of treatment-related ILD, the limiting step is often not the size of the available claims database but the misclassification that erodes the very signal under surveillance. Replacing rule-based claims phenotyping algorithms with zero-shot LLM-based alternatives transforms a high-PPV, low-yield classifier into one that recovers many more patients at the same PPV, converting a classifier that can render drug-safety associations statistically invisible into one that makes them detectable in cohorts of feasible size. Pending external validation, this offers a practical route to more sensitive post-marketing surveillance of the pulmonary toxicity of immune checkpoint inhibitors and antibody-drug conjugates. More broadly, we hypothesise that this advantage is not specific to ILD: zero-shot LLM phenotyping may recover drug-safety signal for other outcomes that diagnosis codes capture poorly but that are well documented in clinical text, a possibility that warrants dedicated evaluation.