Skip to main navigation Skip to search Skip to main content

Multi-Granularity Language-Guided Training for Multi-Object Tracking

  • Northwestern Polytechnical University Xian
  • Mohamed Bin Zayed University of Artificial Intelligence
  • Tianjin University
  • Khalifa University of Science and Technology

Research output: Contribution to journal › Article › peer-review

7 Scopus citations

Abstract

Most existing multi-object tracking methods typically learn visual tracking features via maximizing dis-similarities of different instances and minimizing similarities of the same instance. While such a feature learning scheme achieves promising performance, learning discriminative features solely based on visual information is challenging especially in case of environmental interference such as occlusion, blur and domain variance. In this work, we argue that multi-modal language-driven features provide complementary information to classical visual features, thereby aiding in improving the robustness to such environmental interference. To this end, we propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity (scene-and instance-level) and combines it with standard visual features to obtain discriminative representations. To develop LG-MOT, we annotate existing MOT datasets with scene-and instance-level language descriptions. We then encode both scene-and instance-level language information into high-dimensional embeddings, which are utilized to guide the visual features during training. At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions. Extensive experiments on three benchmarks, MOT17, DanceTrack and SportsMOT, reveal the merits of the proposed contributions leading to state-of-the-art performance. On the DanceTrack test set, our LG-MOT achieves an absolute gain of 2.2% in terms of target object association (IDF1 score), compared to the baseline using only visual features. Further, our LG-MOT exhibits strong cross-domain generalizability. Source code and pre-trained models are available at https://github.com/WesLee88524/LG-MOT.

Original languageEnglish
Pages (from-to)11001-11012
Number of pages12
JournalIEEE Transactions on Circuits and Systems for Video Technology
Volume35
Issue number11
DOIs
StatePublished - 2025

Keywords

  • Multi-object tracking
  • cross-domain generalizability
  • language-guided features

Fingerprint

Dive into the research topics of 'Multi-Granularity Language-Guided Training for Multi-Object Tracking'. Together they form a unique fingerprint.

Cite this