A comprehensive benchmarking study of protein structure alignment tools based on downstream task performance

In this study, we investigate the performance of nine protein structure alignment tools by analyzing the influence of alignment results in accomplishing three downstream biological tasks: homology detection, phylogeny reconstruction, and function inference. These tools include (1) traditional sequential methods using both 3D and 2D structure representations, (2) non-sequential methods, (3) flexible methods, and (4) deep-learning methods. Canonical sequence alignment methods Needleman-Wunsch algorithm and BLASTp are used as baseline methods. We show that accuracies in downstream tasks can be uncorrelated with alignment quality metrics such as TM-score and RMSD, highlighting the discrepancy between the alignment results and the purposes of using them. We identify scenarios where structure alignment results outperform sequence alignment results. In homology detection, structure-based methods are much better than sequence alignment. In phylogeny reconstruction, structure-based methods generally outperform sequence-based methods on the filtered dataset with proteins sharing low sequence similarity. Moreover, we show that structure information improves the overall performance of these tools when used together with sequence information in phylogeny reconstruction and function inference. We also test the running time and CPU/GPU memory consumption of these tools for a large number of queries. Our study suggests that biological problems that were previously addressed with sequence-based methods using only sequence information could be further improved by using structure information alone or using both sequence and structure information. The trade-off between task accuracy and speed is the major consideration in developing new alignment tools for downstream tasks. We recommend both TMalign and KPAX for these tasks because of their good balance between running time and memory consumption, and relatively good and stable accuracy performance in downstream tasks. In tasks that require a large number of pairwise comparisons, such as homology detection and function inference, traditional methods outperform DL methods at the cost of long running time, and Foldseek is the best choice to achieve relatively high accuracy in a reasonable time.