Temporally validated clinical trial outcome prediction from public registry text: a leakage-audited comparison of biomedical transformers and classical NLP

Background: Public clinical trial registries contain information about study design, populations, interventions, and endpoints that may help predict trial outcomes. This study evaluated whether a language model trained using automated Clinical Trial Outcome Dataset (CTOD) probabilities as its only training labels could predict neuroscience trial outcomes. Methods: Phase-specific models were trained on ClinicalTrials.gov records for trials completed before 2019 across all indications. CTOD probabilities were used as continuous training labels, and models were evaluated on post-2019 neuroscience trials using CTOD’s manually curated 2020–2024 outcomes. Inputs were limited to 16 public registry fields, excluding identifiers, dates, status and results modules, the derived CombinedPhases field, and targets. Fixed rules removed explicit descriptions of trial results. BioBERT was selected in a Phase 2 pilot and fixed as the primary model, with BiomedBERT and three matched term frequency–inverse document frequency (TF-IDF) baselines as comparators. Phase 2 remained developmental because its pilot human-reference outcomes had been inspected. Predictions were averaged across five seeds, and uncertainty was estimated from 5,000 bootstrap samples of unique trials. Results: Evaluation included 939 Phase 1, 1,047 Phase 2, and 681 Phase 3 trials. BioBERT achieved areas under the receiver operating characteristic curve (ROC-AUC) of 0.669 (95% confidence interval 0.626 to 0.712), 0.708 (0.675 to 0.741), and 0.719 (0.678 to 0.758), respectively. Corresponding areas under the precision-recall curve (PR-AUC) were 0.431, 0.585, and 0.760, compared with outcome prevalence of 0.230, 0.365, and 0.602. In Phase 3, BiomedBERT reached ROC-AUC 0.745, exceeding BioBERT by 0.027 (Holm-adjusted p=0.026), while the soft-label logistic model reached 0.744. BioBERT Brier skill was negative in Phases 1 and 2 and positive in Phase 3. The full experiment took 3.5 hours on one RTX 5000 Ada graphics processing unit (GPU), at a recorded GPU cost of US$1.19. Conclusions: Training on automated CTOD labels was sufficient to support outcome ranking in later trials using public registry text. Publicly available data allow researchers to reproduce the approach with modest computing resources and without subscription trial databases.