Autonomous generalizability auditing of scientific foundation models

Read the full article See related articles

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

Determining where scientific foundation models generalize beyond their development data is essential for reliable use, but aggregate benchmarks obscure performance limits and rigorous audits are difficult to conduct routinely. Here, we introduce AutoSPECTRA, a coding-agent protocol for systematic generalizability auditing. To inform its design, we reviewed 38 studies across five biological domains and evaluated coding agents, which generated hypotheses and empirical analyses but applied inconsistent standards for judging evidence. We then structured AutoSPECTRA around measurable generalization axes, prespecified analyses, controlled comparisons and independent confirmation. Across four biological foundation models, audits linked performance to protein sequence composition and relatedness to pretraining data (ESMFold2), representation of perturbations in training (STATE), effective spatial resolution (CONCH), and residue-level genomic, evolutionary and structural context (AlphaMissense). Agents also evaluated alternative explanations and rejected unsupported hypotheses. AutoSPECTRA provides a systematic approach to discovering and testing the limits of scientific foundation models.

Article activity feed