A Chemically-Aware Validation Framework for Benchmarking Large Language Models in Materials Synthesis Planning

The rapid integration of large language models (LLMs) into chemistry demands rigorous, domain-specific evaluation metrics that transcend traditional natural language processing (NLP) benchmarks. We introduce a quantitative verification framework to assess the scientific reliability of AI-generated synthesis protocols. This framework integrates two complementary indicators: a framework score evaluating the chemical rationality of synthesis logic and a weighted detail score quantifying the accuracy of experimental parameters. Applied to the synthesis of single-atom catalysts (SACs), it not only establishes a benchmark for automated synthesis generation but also, for the first time, quantifies the gap between conceptual soundness and parameter precision in LLM outputs. Crucially, our analysis reveals that abstract reasoning inherited from broad pretraining, rather than domain-specific stylistic adaptation, is the decisive factor determining scientific accuracy. This insight offers broader implications for the “AI for Science” paradigm. Beyond advancing SAC design, our framework provides a validated “generation-evaluation-optimization” loop that underpins the development of trustworthy autonomous synthesis agents.