Image Fusion Via Mutual Information Maximization for Semantic Segmentation in Autonomous Vehicles

Recognizing and understanding various objects in visual information is paramount for ensuring safe and efficient autonomous navigation, especially in challenging environmental conditions. However, relying solely on single-modal data to perceive information, such as visible images, can compromise the quality and reliability of the extracted information, posing potential risks to autonomous driving systems. To address this challenge, we present a novel fusion method based on mutual information theory in semantic segmentation tasks for secure and efficient autonomous vehicles. The proposed method involves two primary components, i.e., multispectral fusion representation module (FRM) and semantic segmentation module (SSM). To optimize the FRM, we establish a unified quality representation for feature fusion by incorporating an image restoration mechanism, enhancing autonomous driving systems' overall performance and adaptability in complex environments. Meanwhile, we employ mutual information maximization to capture the interimage relations among pixels from the same semantic content, which is achieved by leveraging the semantic representation learned by the SSM in a high-dimensional feature space. Experimental results and comparisons with famous fusion approaches and segmentation models on four public datasets validate our method's effectiveness, robustness, and overall superiority.