From raw footage to behaviour: an automated pipeline for activity monitoring and behaviour identification in the wild

Studies of animal behaviour usually rely on direct observations or manual annotations of video recordings for data collection. However, such methods can be very time-consuming and inefficient, leading to sub-optimal sample sizes. Recent advances in deep-learning have shown great potential to overcome such limitations. Nonetheless, for behavioural recognition, solutions currently available are mostly focused on laboratory and/or standardized settings.

Here, we present a pipeline to guide researchers in building their own behavioural recognition system from their video data. Our pipeline is replicable and especially useful for studies in the wild. We take advantage of over 12215hs of manually annotated behaviours of sociable weavers ( Philetarius socius ). As behaviour classifiers we used long-short term neural networks (LSTMs) known to be suitable to classify temporal sequences of data such as video-based behavioural data.

LSTMs allowed us to: 1) monitor nest activity by detecting the birds’ presence and simultaneously classifying the type of trajectory: i.e., nest-chamber entrance or exit; and 2) identify the respective behaviour performed: provisioning, building or aggression. With our pipeline we reduced the behavioural annotation error by 21%, while increasing by eight times the analyses speed.

This work demonstrates how previously analysed long-term data can be used to develop highly-efficient automated behavioural analyses, whereas this pipeline is easily adaptable to other species and behaviours, requiring only labelled video data.