Phenotypic AI-based design of cell-specific small molecule cytotoxics

Drug discovery is being transformed by artificial intelligence, which enables the exploration of vast chemical spaces and the generation of novel compounds with tailored properties. While most generative approaches focus on optimizing physicochemical features or protein-binding affinities, their application to phenotypic drug discovery remains limited. Here we present an integrated framework to design small molecules with selective cytotoxic effects in pancreatic cancer cells that combines high-throughput cytotoxicity screening, bioactivity signature–based machine learning predictors, and reinforcement learning–driven generative models. We screened over 11,000 compounds across six pancreatic cancer and two control cell lines, confirming 392 hits, and used these data to train cell line–specific classifiers that outperformed fingerprint-based models. We embedded these predictors into the REINVENT platform to generate molecules with desired cytotoxicity profiles, including challenging cases of distinguishing between pancreatic cancer lines with highly similar molecular backgrounds. We generated between 37 and 137 high- scoring candidates in seven different cell selectivity exercises, which were structurally distant from the active compounds in the training set. We tested 45 close analogues to the AI-designed molecules, and 20 of them showed the desired cell-specific cytotoxic effects. Indeed, we validated at least one compound in 5 of the 7 exercises tested, showing over a large increase in hit discovery rate compared to the HTS. Additionally, we benchmarked our strategy against direct predictions on a library of over 150,000 compounds, skipping the generative step, finding that the fraction of hits, and more importantly the dose-response validation, is markedly higher in the AI- designed molecules. Overall, our study demonstrates the feasibility of combining predictive and generative AI methods to design molecules with complex phenotypic outcomes in a target- agnostic manner, paving the way for next-generation phenotype-driven drug discovery strategies.