A systematic DNN weight pruning framework based on symmetric accelerated stochastic ADMM

Weight pruning is widely employed in compressing Deep Neural Networks (DNNs) because of the increasing computation and storage requirement. However, related work failed to efficiently combine the structure of the DNN loss function with the Alternating Direction Method of Multipliers (ADMM). This paper presents a systematic weight pruning framework of DNNs using the advanced symmetric accelerated stochastic ADMM (SAS-ADMM). Specifically, the weight pruning problem of DNNs is formulated as an optimization problem that consists of the DNN loss function and a L1 regularization term. SAS-ADMM is widely used to solve the problem by dividing it into two small-dimensional and relatively easier subproblems. Besides, an optimizer based on SAS-ADMM is presented to make the DNNs after pruning converge. Experimental results demonstrate that our method achieves a faster convergence rate in a better or similar weight pruning rate than previous work. For the CIFAR-10 data set, our method reduces the number of ResNet-32 and ResNet-56 parameters by a factor of 6.61× and 9.93 × while maintaining accuracy. In similar experiments of AlexNet on the ImageNet data set, we achieve 20.9× weight reduction, which only takes half of the time compared with prior work.