DLC2Action: A Deep Learning-based Toolbox for Automated Behavior Segmentation

While expert biologists can annotate complex behaviors from video data, the process remains tedious and time-consuming, creating a bottleneck for efficient behavioral analysis. Here, we present DLC2Action, an open-source Python toolbox that enables automatic behavior annotation from video or estimated 2D/3D pose tracking data. DLC2Action integrates multiple state-of-the-art deep learning architectures, optimized for action segmentation and supports self-supervised learning (SSL) to leverage unlabeled data, boosting performance with limited labeled datasets. Its robust implementation enables efficient hyperparameter optimization, customizable feature extraction, and data handling. We also standardized eight benchmarks and evaluated DLC2Action on five animal behavior datasets, which comprise common behavioral tests in neuroscience, and four human datasets. Overall these datasets span a wide range of contexts from standard laboratory studies to naturalistic cooking. DLC2Action reached strong performance across those benchmarks. To further showcase the tool's versatility, we applied it to Atari gameplay data and found that in certain games the players' eye movements consistently predict their button presses across different subjects. Furthermore, DLC2Action features an intuitive graphical user interface (GUI) for streamlined behavior annotation, active learning, and assessment of model predictions. Diverse pose, video, and annotation formats are supported. Lastly, DLC2Action is modular and thus designed for extensibility, allowing users to integrate new models, dataset features, and methods. The code and benchmarks are available at: https://github.com/amathislab/DLC2action