跳到主要导航 跳到搜索 跳到主要内容

MFD-KD: Multi-Scale Frequency-Driven Knowledge Distillation

  • Tao Dai
  • , Rongliang Huang
  • , Hang Guo
  • , Jinbao Wang
  • , Bo Li
  • , Zexuan Zhu
  • Shenzhen University
  • Tsinghua University
  • Northwestern Polytechnical University Xian

科研成果: 期刊稿件 › 文章 › 同行评审

8 引用 (Scopus)

摘要

Knowledge Distillation (KD), which focuses on transferring semantic knowledge from a parameter-heavy teacher network to a more compact student network, has been widely and successfully used across plenty of computer vision tasks like image/video recognition and segmentation. Unlike traditional KD methods that primarily focus on the consistency of intermediate features in the spatial domain, we propose a novel Multi-scale Frequency-Driven Knowledge Distillation (MFD-KD) framework, which emphasizes the utilization of information in the frequency domain. Specifically, our method adopts Fast Fourier Transform (FFT) to shift intermediate feature maps of spatial domain into the corresponding frequency domain, enabling our approach to extract crucial high-and low-frequency information both inside and outside the frequency layer's square center, while also minimizing interference from non-semantic information, such as noise. Furthermore, we have developed a logic layer similarity distillation loss to further leverage the category information of the teacher network, thereby enhancing the performance of the student network. Extensive experiments with various architectures including Convolution Neural Network (CNN)-based and Transformer-based networks on public datasets, such as CIFAR100, ImageNet, and MSCOCO, have shown that our method can significantly improve the performance of these models. The code is available at https://github.com/ronglianghuang/MFD-KD.

源语言英语
页(从-至)563-574
页数12
期刊Big Data Mining and Analytics
卷8
期3
DOI
出版状态已出版 - 2025
已对外发布是

学术指纹

探究 'MFD-KD: Multi-Scale Frequency-Driven Knowledge Distillation' 的科研主题。它们共同构成独一无二的学术指纹。

引用此