Weakly Correlated Multimodal Domain Adaptation for Pattern Classification

Multimodal domain adaptation (MMDA) aims to transfer knowledge across different domains that contain multimodal data. Current methods typically assume that both the source and target domains have paired multimodal data with the same modalities, allowing for direct knowledge transfer between corresponding types of data. However, in certain applications, the source domain benefits from advanced sensors and equipment, capturing more modalities than those available in the target domain. As a result, the information from the source modalities may not strongly align with that of the target modalities. This weak correlation hinders the effective utilization of all source data for the target domain. To address this challenge, we propose a weakly correlated multimodal domain adaptation (WCMMDA) method for pattern classification. WCMMDA is designed to acquire the modality-independent and category-related knowledge from the source domain, enabling the full utilization of available source modalities for effective knowledge transfer. Specifically, modality-invariant features are first extracted from the multimodal data to bridge the heterogeneity gap within each domain. Subsequently, domain-invariant features are further learned from these modality-invariant features to align the feature distributions across the source and target domains. A source-specific classifier is employed here, which predicts pseudo-labels for the target data and enables the feature extractor to explore category-related information in source features. Finally, a target-specific classifier is trained using the pseudolabeled target data, where highly reliable pseudolabels are selected based on confidence to improve classification performance. Extensive experiments are performed on the real-world multimodal datasets to demonstrate the superiority of WCMMDA.