Benchmarking AI Models for In Silico Gene Perturbation of Cells

Understanding perturbations at the single-cell level is essential for unraveling cellular mechanisms and their implications in health and disease. The growing availability of biological data has driven the development of a variety of in silico perturbation methods designed for single-cell analysis, which offer a means to address many inherent limitations of experimental approaches. However, these computational methods are often tailored to specific scenarios and validated on limited datasets and metrics, making their evaluation and comparison challenging. In this work, we introduce a comprehensive benchmarking framework to systematically evaluate in silico perturbation methods across four key scenarios: predicting effects of unseen perturbations in known cell types, predicting effects of observed perturbations in unseen cell types, zero-shot transfer to bulk RNA-seq of cell lines, and application to real-world biological cases. For each scenario, we curated diverse and abundant datasets, standardizing them into flexible formats to enable efficient analysis. Additionally, we developed multiple metrics tailored to each scenario, facilitating a thorough and comparative evaluation of these methods. Our benchmarking study assessed 10 methods, ranging from linear baselines to advanced machine learning approaches, across these scenarios. While some methods demonstrated surprising efficacy in specific contexts, significant challenges remain, particularly in zero-shot predictions and the modeling of complex biological processes. This work provides a valuable resource for evaluating and improving in silico perturbation methods, serving as a foundation for bridging computational predictions with experimental validation and real-world biological applications.