Benchmarking and behavioral characterization of LLM agents for protein design
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (Arcadia Science)
Abstract
Large language models (LLMs) are increasingly deployed as agents for scientific discovery, but standardized frameworks for evaluating their performance and behavior in scientific workflows are lacking. Protein design provides a demanding test case because modern workflows combine stochastic generative models, structure prediction systems, and physics-based evaluation tools that require extensive candidate exploration and filtering. Here we introduce BioDesignBench, a benchmark of 76 expert-curated protein design tasks spanning antibodies, enzymes, fluorescent proteins, binders, and scaffolds, together with human and non-LLM baselines and behavioral metrics derived from tool-use traces. We evaluate four frontier LLM agents across diverse protein design workflows and find that the strongest agents surpass deterministic hardcoded pipelines but consistently underperform expert practice. Decomposing agent behavior along orthogonal axes of tool coverage and evaluation depth, we localize the gap to evaluation rather than tool selection: agents generally pick appropriate tools but score each candidate against only a narrow set of metrics, rarely compare alternatives, and terminate exploration prematurely. Guided workflows improve tool coverage but not evaluation depth. A compute-matched forced-depth intervention that requires agents to evaluate every candidate across multiple complementary metric categories (structure, interface, physics, and learned affinity) substantially improves performance and rules out generic scaffolding effects, demonstrating that the gap is behavioral rather than a fundamental capability constraint. All evaluations are performed in silico. We release BioDesignBench, open-source reference agents, and a public leaderboard as a community resource for evaluating and improving AI agents for protein engineering.
Article activity feed
-
This is genuinely useful and timely work. It does a nice job localizing the problem (shallow evaluation) and then testing that diagnosis with a real intervention. Much of the current industry conversation around LLM protein design workflows centers on model selection, so having evidence that the operative bottleneck is actually approach and orchestration — not which model you pick — is a valuable counterpoint. A few points of feedback:
- Figure 1b: there's a typo under the PLM-guided section — "sequence" is misspelled.
- LLM-judge scoring (28 of 100 points): If the LLM judges aren't assessing biophysical quality, what exactly are they scoring? Spelling that out more explicitly would help. I'm also unclear on how you arrived at 28 as the LLM-judged share specifically, and what work went into the weighting calibration— both across the six …
This is genuinely useful and timely work. It does a nice job localizing the problem (shallow evaluation) and then testing that diagnosis with a real intervention. Much of the current industry conversation around LLM protein design workflows centers on model selection, so having evidence that the operative bottleneck is actually approach and orchestration — not which model you pick — is a valuable counterpoint. A few points of feedback:
- Figure 1b: there's a typo under the PLM-guided section — "sequence" is misspelled.
- LLM-judge scoring (28 of 100 points): If the LLM judges aren't assessing biophysical quality, what exactly are they scoring? Spelling that out more explicitly would help. I'm also unclear on how you arrived at 28 as the LLM-judged share specifically, and what work went into the weighting calibration— both across the six rubric categories, and between the LLM-judge and algorithmic components within Approach and Orchestration.
- Compute/time cost: does the rubric account for the computational or wall-clock cost of the design process anywhere? Given how compute-intensive protein design already is, an agent's efficiency (not just its final score) seems like an important axis to capture, especially if this framework is meant to inform real-world LLM choice.
- Figure 2d: the heatmap coloring doesn't seem to track the underlying values consistently — a score of three, for instance, appears in different shades across and within columns, and some columns (Novelty, Diversity) look uniformly lighter than others despite covering a similar value range. If this comes down to unreported decimal precision, either showing the decimals or adjusting the color scale would help avoid a misleading read at a glance. Overall, this is a well-designed benchmark and a clear, actionable diagnosis of where LLM agents may fall short of expert practice — nice work.
-
