Selective Alignment Transformer for Partial-Set Remote Sensing Image Cross-Scene Classification

Cross-scene classification aims to transfer knowledge acquired from a label-rich source domain to an unlabeled target domain with a distribution shift. With the large amount of available remote sensing data from diverse satellite platforms, it prompts us to utilize the knowledge from extensive datasets to address target tasks in small-scale domains, known as partial domain adaptation (PDA). However, the PDA setting poses significant challenges for remote sensing scene images. Existing methods often fail to sufficiently explore both task-specific and transferable knowledge across domains based on the representations of entire samples, potentially resulting in the amplification of negative transfer brought by irrelevant knowledge. To address this, we propose a new selective alignment transformer (SAT) designed to distinguish transferable and untransferable knowledge across domains for cross-scene classification in RSIs under the PDA scenario. Specifically, a new bi-level reweighting strategy that incorporates transferability-aware patch selection and class-wise reweighting is developed to emphasize the transferable image patches and classes. Based on the aforementioned reweighting strategy, we further introduce a patch-weighted maximum mean discrepancy (PMMD) loss, which selectively aligns the distributions from the patch-level perspective, facilitating the learning of transferable domain-invariant representations. The experimental results of SAT demonstrate its effectiveness and superiority in addressing this practical domain adaptation (DA) task, outperforming state-of-the-art methods in PDA tasks on four datasets.