Prototype Decoupled Knowledge Distillation

Knowledge distillation (KD) has emerged as a powerful technique for transferring knowledge from large, complex teacher models to smaller, more efficient student models. However, current KD methods primarily concentrate on mimicking instance-level predictions or feature representations, often overlooking the crucial role of class-level semantic structure in guiding effective knowledge transfer. This paper introduces Prototypical Decoupled Knowledge Distillation (PDKD), a novel framework designed to bridge the gap between instance-specific and class-discriminative knowledge by leveraging class prototypes. PDKD incorporates a prototype-aware supervision module that distills global class characteristics by aligning student predictions with both instance-level and prototype-based outputs from the teacher. This module dynamically harmonizes the logit scales of these two targets, effectively addressing the model size mismatches between teacher and student. Furthermore, a feature discrepancy alignment module is proposed to enforce consistency between the teacher and student in how they modulate features between the learned prototypes and individual samples. This alignment preserves the structural relationships between classes. By effectively unifying hierarchical class semantics with instance-level learning, PDKD establishes a new paradigm for training compact yet highly discriminative models. Extensive experiments on CIFAR-100 and ImageNet showcase the superior performance of PDKD compared to existing state-of-the-art methods.