TY - JOUR
T1 - What to Keep and How to Drop
T2 - Informative Token Pruning for Efficient Vision Transformers
AU - Cheng, Jiacheng
AU - Yao, Xiwen
AU - Yuan, Xiang
AU - Cheng, Gong
N1 - Publisher Copyright:
© 1991-2012 IEEE.
PY - 2026
Y1 - 2026
N2 - Vision Transformers (ViTs) are computationally demanding for resource-constrained deployment. Existing token-reduction methods mainly focus on identifying important tokens, while paying less attention to preserving useful information carried by discarded tokens. In this paper, we propose ToPro (Token Pruning with Informative Propagation), a lightweight token-pruning framework that derives token retention scores from attention statistics through a learnable mapping, avoiding complex auxiliary predictors. To mitigate information loss, ToPro propagates informative cues from discarded tokens to semantically related retained tokens. We further introduce non-linear augmentation to alleviate token-score homogenization in deeper layers and maintain token discriminability. Extensive experiments across six configurations from three representative ViT families (ViT, DeiT, and LV-ViT) show favorable accuracy–efficiency trade-offs on ImageNet-1K, NWPU-RESISC45, and AID. On the 19-task VTAB-1K benchmark, ToPro consistently achieves the highest Group Mean among pruning methods for all six backbones. ToPro reduces ViT-Base FLOPs by 43.55% with a 0.82-percentage-point accuracy drop on ImageNet-1K. Beyond classification, ToPro achieves the highest accuracy with the lowest FLOPs among the evaluated token-pruning methods on COCO object detection and ADE20K semantic segmentation.
AB - Vision Transformers (ViTs) are computationally demanding for resource-constrained deployment. Existing token-reduction methods mainly focus on identifying important tokens, while paying less attention to preserving useful information carried by discarded tokens. In this paper, we propose ToPro (Token Pruning with Informative Propagation), a lightweight token-pruning framework that derives token retention scores from attention statistics through a learnable mapping, avoiding complex auxiliary predictors. To mitigate information loss, ToPro propagates informative cues from discarded tokens to semantically related retained tokens. We further introduce non-linear augmentation to alleviate token-score homogenization in deeper layers and maintain token discriminability. Extensive experiments across six configurations from three representative ViT families (ViT, DeiT, and LV-ViT) show favorable accuracy–efficiency trade-offs on ImageNet-1K, NWPU-RESISC45, and AID. On the 19-task VTAB-1K benchmark, ToPro consistently achieves the highest Group Mean among pruning methods for all six backbones. ToPro reduces ViT-Base FLOPs by 43.55% with a 0.82-percentage-point accuracy drop on ImageNet-1K. Beyond classification, ToPro achieves the highest accuracy with the lowest FLOPs among the evaluated token-pruning methods on COCO object detection and ADE20K semantic segmentation.
KW - Token pruning
KW - efficient ViT
KW - model compression
KW - token propagation
UR - https://www.scopus.com/pages/publications/105052615821
U2 - 10.1109/TCSVT.2026.3739834
DO - 10.1109/TCSVT.2026.3739834
M3 - 文章
AN - SCOPUS:105052615821
SN - 1051-8215
JO - IEEE Transactions on Circuits and Systems for Video Technology
JF - IEEE Transactions on Circuits and Systems for Video Technology
ER -