TY - GEN
T1 - DeSD
T2 - 25th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2022
AU - Ye, Yiwen
AU - Zhang, Jianpeng
AU - Chen, Ziyang
AU - Xia, Yong
N1 - Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.
PY - 2022
Y1 - 2022
N2 - Self-supervised learning (SSL), enabling advanced performance with few annotations, has demonstrated a proven successful in medical image segmentation. Usually, SSL relies on measuring the similarity of features obtained at the deepest layer to attract the features of positive pairs or repulse the features of negative pairs, and then may suffer from the weak supervision at shallow layers. To address this issue, we reformulate SSL in a Deep Self-Distillation (DeSD) manner to improve the representation quality of both shallow and deep layers. Specifically, the DeSD model is composed of an online student network and a momentum teacher network, both being stacked by multiple sub-encoders. The features produced by each sub-encoder in the student network are trained to match the features produced by the teacher network. Such a deep self-distillation supervision is able to improve the representation quality of all sub-encoders, including both shallow ones and deep ones. We pre-train the DeSD model on a large-scale unlabeled dataset and evaluate it on seven downstream segmentation tasks. Our results indicate that the proposed DeSD model achieves superior pre-training performance over existing SSL methods, setting the new state of the art. The code is available at https://github.com/yeerwen/DeSD.
AB - Self-supervised learning (SSL), enabling advanced performance with few annotations, has demonstrated a proven successful in medical image segmentation. Usually, SSL relies on measuring the similarity of features obtained at the deepest layer to attract the features of positive pairs or repulse the features of negative pairs, and then may suffer from the weak supervision at shallow layers. To address this issue, we reformulate SSL in a Deep Self-Distillation (DeSD) manner to improve the representation quality of both shallow and deep layers. Specifically, the DeSD model is composed of an online student network and a momentum teacher network, both being stacked by multiple sub-encoders. The features produced by each sub-encoder in the student network are trained to match the features produced by the teacher network. Such a deep self-distillation supervision is able to improve the representation quality of all sub-encoders, including both shallow ones and deep ones. We pre-train the DeSD model on a large-scale unlabeled dataset and evaluate it on seven downstream segmentation tasks. Our results indicate that the proposed DeSD model achieves superior pre-training performance over existing SSL methods, setting the new state of the art. The code is available at https://github.com/yeerwen/DeSD.
KW - Deep self-distillation
KW - Medical image segmentation
KW - Self-supervised learning
UR - http://www.scopus.com/inward/record.url?scp=85139030848&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-16440-8_52
DO - 10.1007/978-3-031-16440-8_52
M3 - 会议稿件
AN - SCOPUS:85139030848
SN - 9783031164392
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 545
EP - 555
BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2022 - 25th International Conference, Proceedings
A2 - Wang, Linwei
A2 - Dou, Qi
A2 - Fletcher, P. Thomas
A2 - Speidel, Stefanie
A2 - Li, Shuo
PB - Springer Science and Business Media Deutschland GmbH
Y2 - 18 September 2022 through 22 September 2022
ER -