Illumination-Aware Infrared and Visible Image Fusion for Nighttime Object Tracking

The change of illumination has a great impact on the imaging effect of visible image. In the tracking scene of bad illumination or weather conditions, visible image cannot effectively and comprehensively describe the imaging scene. It is important to consider the effect of illumination change in the process of feature extraction and fusion, and fuse complementary information from visible modality and thermal infrared modality to provide a comprehensive description of the scene. Aiming to enhance the performance of nighttime object detection and tracking, this paper designs an illumination-aware module, which utilizes the illumination information contained in the visible image. The module generates the fusion weights for the visible and thermal infrared images. Then the multi-modal fusion algorithm based on the module for object tracking is proposed. Extensive experiments have shown that the proposed illumination-aware multi-modal fusion effectively improves the performance of object tracking in the nighttime environment.