Low-latency multicamera 3D tracking of insects with Braid
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Advances in camera technology and computer vision techniques have allowed researchers to track animals in 3D in ways which previously were difficult or impossible. Many such 3D tracking tools make use of multiple cameras, but unfamiliarity with the principles and technology involved can make it difficult to employ such techniques. In this protocol, we describe Braid, open-source software for live, multi-camera 3D tracking of insects. Using background-subtraction, Braid performs detection of objects without requiring the use of physical markers affixed to the insect. Braid constructs low-latency 3D position estimates using Kalman filtering and nearest neighbor data association. We document in detail the process of tracking freely flying bees within a flight arena using Braid. This protocol includes instructions on installation, configuration of cameras, setup, calibration, and operation. Within the system described here, we demonstrate that Braid can achieve position estimates accurate to <1 millimeter (within a 0.3 cubic meter volume). These factors make Braid suitable for tracking small, fast-flying animals like insects. Braid’s low latency allows live tracking, removing the necessity to collect large video files and making it suitable for integration in closed loop systems such as virtual reality. Code is available at https://github.com/strawlab/strand-braid
Key features
Guides the reader through setup and use of Braid multi-camera acquisition and tracking software.
Braid allows low latency, marker-free real-time 3D tracking of multiple objects, with a web-based GUI and integration with Rerun visualization software.
Braid has all necessary calibration tools incorporated within it. We provide detailed instructions on how to calibrate a Braid-multicamera system.
Braid is useful for 3D tracking of animals, robots, or other moving objects, particularly when these objects are small such as insects.
This protocol is used in: The components of the Braid system itself described here (cameras, PC, switch, etc.) was used as the ‘static camera’ system in Vo-Doan et al. [1]. Braid itself has also been used in: [2–8]. The algorithms used in Braid are similar or identical to those of its predecessor software Flydra [9] that has been used in: [10–19].
Graphical overview
Graphical Overview: Schematic of 3D tracking with Braid described here. Sequence of events that occur at each timepoint are described left to right. Red lines indicate the flow of information. Initially the flight of an insect, here a honey bee (inset), is observed by a series of synchronized cameras. The bee’s full flight trajectory to the yellow artificial flower is indicated by green points. Image frames from each camera are acquired and processed online with Strand Camera, with one instance of Strand Camera software running per physical camera. Central images show the image captured for the timepoint indicated by the black triangle on the bee’s flight trajectory. Strand Camera detects the insect in two dimensions. Green dots, visible within magnifications of image frames, indicate the location of these 2D detections. These detections are given to a single instance of the Braid software. Braid then utilizes a Kalman filter based algorithm and camera calibrations to produce estimates of the bee 3D position in real time. The full flight trajectory of the bee looking down the x (red axis), y (green axis), and z (blue axis) axes are shown (panels x, y and z respectively).
