MSPFusion: A feature transformer for multidimensional spectral-polarization image fusion

Spectral images capture reflection characteristics, texture, and spatial details, while polarimetric images provide insights into surface roughness, morphology, and material structure. Fusing these complementary sensing modalities overcomes the limitations of individual perception techniques, offering a richer understanding of the scene. However, traditional fusion methods are typically designed for dual-modal inputs and lack robustness when handling multimodal data. To address this, we propose MSPFusion, a novel feature transformer network for multidimensional spectral-polarization image fusion. Our approach leverages a Feature Multi-head Self-attention (F-MSA) mechanism in the encoder to enhance feature extraction and introduces an entropy-based fusion layer to adaptively integrate spectral and polarization information by maximizing information content. The F-MSA decoder further refines the fused features to produce high-quality integration and visualization of spectral-polarization data. We constructed a hyperspectral-polarization dataset, NWPU-SP, to validate our method. Experimental results show that MSPFusion achieves state-of-the-art performance, ranking first in the following metrics, i.e., entropy (EN), information (MI), standard deviation (SD), quality of saliency (QS), and quality of gradient (QG). All these metrics show a better preservation of information, along with improved clarity and contrast in the fusion results of our proposed method. Additionally, MSPFusion demonstrates strong robustness in infrared–visible image fusion tasks, proving its adaptability across various scenarios. The source code and NWPU-SP dataset are available at https://github.com/tgg-77/MSPFusion.