Attention correction feature and boundary constraint knowledge distillation for efficient 3D medical image segmentation

Incorporating auxiliary modules into the teacher to facilitate feature knowledge transfer often incurs additional training costs, significantly impeding the application of feature-based knowledge distillation in 3D medical image segmentation. This paper explores the non-negative matrix factorization (NMF) theory to illustrate that ResNet-like or UNet-like residual connections and attention mechanisms function as corrections to input features. Based on this discovery, we propose Attention Correction Feature Distillation (ACFD), which comprises Correction Factor Distillation (CFD) and Attention-based Feature Distillation (AFD). In CFD, the teacher's residual correction factor guides the student's learning of the attention correction factor, while in AFD, the teacher's residual-corrected features are used to teach the student's attention-corrected features. Furthermore, the introduction of boundary constraints can enhance the student's boundary perception, but Hard Boundary Distillation (Hard BD) is sensitive to noise. Hence, we introduce Soft Boundary Distillation (Soft BD) to extract soft boundary information by estimating the Hausdorff Distance (HD) and integrate both Hard BD and Soft BD into Boundary Constraint Distillation (BCD). Ultimately, we present the Attention Correction Feature and Boundary Constraint Knowledge Distillation (ACF-BCKD). Comprehensive experiments on three benchmark datasets — BTCV, WORD, and BraTS-23 — demonstrate the state-of-the-art performance of ACF-BCKD. Ablation experiments and visualization results confirm the effectiveness of individual modules.