Privacy-Preserving Large Language Model Deployment for Oncology Registry Abstraction: Structure-Aware Evaluation in a Real-World Clinical Setting
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Structuring oncology clinical notes into registry-grade variables is essential for research and care but remains labour-intensive and error-prone.
Objective
To develop and evaluate a privacy-preserving large language model pipeline for oncology registry abstraction in a real-world clinical setting.
Methods
We deployed an open-source Meta Llama 3.3 70B–based pipeline to extract over 50 variables from 6,700 oncology notes at a cancer centre in Singapore. Data were de-identified locally using a Hide-In-Plain-Sight approach, ensuring no identifiable data left hospital infrastructure. Performance was assessed on 200 randomly sampled notes with adjudicated ground truth. A structure-aware framework classified outputs as correct, missing, spurious, or incorrect.
Results
F1 scores were high across variables, including diagnosis (97.2%), histology (95.8%), stage (92.6%), biomarkers (91.4%), and treatments (88.1%). Transferability testing on 50 external notes showed strong performance for core variables.
Conclusions
Privacy-preserving LLMs can achieve near–human-level accuracy for oncology abstraction, with structure-aware evaluation enabling more clinically meaningful assessment.
