LieOTAlign: A Differentiable Protein Structure Alignment Framework Combining Optimal Transport and Lie Algebra

The comparison of protein structures is fundamental to understanding biological function and evolutionary relationships. Existing methods, while powerful, often rely on heuristic search algorithms and non-differentiable scoring functions, which limits their direct integration into end-to-end deep learning pipelines. This paper introduces LieOTAlign ¹ , a novel and fully differentiable protein structure alignment framework built on the mathematical principles of Lie algebra and Optimal Transport (OT). LieOTAlign represents rigid body transformations within the Lie algebra of SE(3), which intrinsically preserves the geometric validity of rotations and translations during optimization. We formulate the alignment task as an optimal transport problem, seeking the most efficient mapping between two protein structures. This approach leads to a differentiable version of the TM-score, the Sinkhorn score, which is derived from the entropically regularized OT solution computed via the Sinkhorn algorithm. The entire LieOTAlign pipeline is differentiable, enabling the use of gradient-based optimizers like AdamW to maximize structural similarity. Benchmarking against the official TM-align on the RPIC dataset shows that LieOTAlign can identify longer, topologically significant alignments, achieving higher TM-scores. While the current RMSD is higher, LieOTAlign provides a powerful and flexible framework for protein structure alignment, paving the way for its integration into next-generation deep learning models for diverse bioinformatics challenges.