An optimized method for remote sensing image classification distillation

Knowledge distillation is an effective method for enhancing the performance of small neural networks. Existing distillation methods mainly involve extracting deep features from intermediate layers and using spatial matching techniques to enable the student model to learn from the teacher model’s knowledge. However, due to differences in the receptive field scales between large and small neural networks, the feature maps of the student and teacher models represent different regions of the image during the distillation process. Consequently, the student model cannot accurately acquire the knowledge of the teacher model, leading to performance degradation. To this end, we propose the Deformable Fast Transformer approach. Leveraging the inherent symmetry present in remote sensing images, such as airplanes and ships, our method adjusts the flipping angle of the student network’s feature map to mitigate the suboptimal distillation results caused by unequal receptive fields. As our method only involves calculations on the feature map, DFT can be applied to various models. We conducted experiments on models with different backbones, and the results demonstrate that the student network with our method achieves superior performance. For instance, in distillation experiments with ResNet110 as the teacher network and ResNet18 as the student network, the accuracy reached 72.01% on DOTA-v1.0, which are 1% higher than the baseline.