FastTrackTr: Real-Time Multiobject Tracking With Transformers for Real World

Transformer-based multiobject tracking (MOT) methods have attracted significant attention from researchers. However, these Transformer-based models often suffer from suboptimal inference speeds due to their architectural complexities or other inherent issues, rendering them difficult to deploy in practical industrial applications. To address this challenge, we revisited the classic joint detection and tracking (JDT) paradigm and analyzed existing models. Drawing inspiration from Detection Transformer’s (DETR) object queries, which naturally encode object appearance features, we constructed a fast and novel JDT-type MOT framework named FastTrackTr by implementing an efficient interframe information transfer mechanism. This framework integrates three key technological innovations: a cross-decoder mechanism that implicitly incorporates historical trajectory information without requiring additional queries or decoders, a historical encoder and decoder pair for refining and utilizing historical feature representations, and a deterministic fixed-shape architecture that enables seamless TensorRT acceleration. Benefiting from these designs, our approach not only reduces the number of queries required for tracking but also avoids introducing excessive network structures, ensuring model simplicity while maintaining high accuracy. Experimental results show that our method achieves real-time tracking while maintaining state-of-the-art accuracy. On an NVIDIA RTX 4090 with an image size of 1333 × 800, it reaches 62.4 higher order tracking accuracy (HOTA) at 86.6 frames per second (FPS) on the DanceTrack dataset, outperforming other advanced Transformer-based methods. Furthermore, it excels on edge devices, such as the NVIDIA Jetson AGX Orin, where its lightweight variant achieves up to 59.2 FPS on 640 × 640 images, meeting real-time requirements for practical applications.