DIPLOMAT: multi-animal tracking with efficient manual editing

Recent advances in computer vision have enabled the development of automated animal behavior observation tools. Several software packages currently exist for concurrently tracking pose in multiple animals; however, existing tools still face challenges in maintaining animal identities across frames and can demand extensive human oversight and editing. Here we report on DIPLOMAT, a Deep learning-based, Identity-Preserving, Labeled-Object Multi-Animal Tracker, which implements automated algorithms improving tolerance to occlusion and continuity of animal identity over a video, further supplemented by an efficient human interface to help eliminate remaining errors. DIPLOMAT is designed to perform multi-animal tracking by building on the per-frame pose prediction models of two state-of-the-art tools, DeepLabCut and SLEAP. Where other tools immediately take a maximum likelihood estimate from a given video frame, DIPLOMAT splits the probability fields according to the number of tracked animals and then applies an inference method that takes into account across-frame movement and probabilistic distances between body parts. These independent trace probabilities are then preserved for human editing, enabling multiple body parts to be re-tracked across frames with minimal user action. Testing with a standardized and independently tracked dataset of 3-mouse videos shows DIPLOMAT's automated components alone can reduce identity swaps by >75\%. DIPLOMAT code and documentation are available at https://diplomattrack.org/