Interactive Fusion of Embodied Multimodal Data for Smart Cockpit Perception

Multimodal data fusion, as a key supporting technology for embodied intelligence, plays an important role in embodied interaction tasks such as autonomous driving, aircraft visual navigation, and military platforms. However, the traditional fusion paradigms face problems such as the difficulty in ensuring the fusion quality, the insufficient model generalization, and the lack of interpretability in decision-making processes in complex interactive environments. Therefore, this paper proposes a visual language-guided embodied multimodal data interactive fusion method for the multispectral environment perception and driver gaze interaction scenarios of intelligent agent vehicles in the military field. On the one hand, an interpretable multi-scale interactive fusion model is designed based on the embodied cognition theory and the Kolmogorov Arnold representation theorem. It is used to jointly model the multispectral environmental data and driver eye movement data, achieving the fusion enhancement of the focused area. On this basis, the semantic prior and representation ability of the visual language pretrained large model are introduced to optimize the multimodal fusion small model. The experimental results show that the proposed method overcomes the problems of difficult generalization and insufficient interpretability of multimodal fusion in complex interactive environments to a certain extent, and significantly improves the clarity of fused images, providing technical support for the application of interpretable human-machine interaction and embodied intelligence in transparent smart cockpits.