The ADVANCE toolkit: automated descriptive video annotation in naturalistic child environments

Video recordings are commonplace for observing human and animal behaviours, including inter-individual interactions. Analyses for clinical applications remain particularly cumbersome, requiring human-based annotation that is time-consuming, bias-prone, and cost-ineffective. Attempts of machine learning to address these limitations still oftentimes require highly standardised environments, scripted scenarios, and forward-facing individuals. Here, we provide the ADVANCE toolkit - an automated video annotation pipeline. The versatility of ADVANCE is demonstrated with schoolchildren and adults in an unscripted clinical setting that included 2-5 individuals, occlusions, and large variations in actions. We accurately detected each individual, tracked them simultaneously throughout the duration of the recording, estimated the position of their skeleton joints, and labelled their poses. By resolving challenges of manual annotation, we radically enhance the ability to extract information from video recordings across different scenarios and settings. This toolkit reduces clinical workload and enhances ethological validity, offering scalable solutions for behaviour analyses in naturalistic contexts.