Dynamic Activation Binarization for Accurate Network Compression

Binary neural networks (BNNs), of which both weights and activations are binarized into 1 bit, have shown great potential in network compression. However, due to the limited expressive capacity of a single bit, they suffer from severe performance degradation. To mitigate this problem, we present a dynamic activation binarization method for BNNs. In contrast to existing BNN methods which utilizes Sign and ReLU functions with fixed thresholds for activation binarization, we introduce auxiliary controllers to dynamically generate thresholds for each test image based on its content, which empowers the BNN to flexibly exploit the characteristics of each image to achieve optimal activation binarization, thus increasing the expressive capacity of BNN. Moreover, we construct each controller using a light-weight structure, which only slightly increases the computational and storage complexity over the baseline, but improves the performance obviously. Experiments on two standard classification benchmarks verify the superiority of the proposed method over many state-of-the-arts. In addition, the proposed BNN also has excellent performance on other computer vison related tasks, which even obtains comparable performance with a SOTA 4-bit network on super-resolution task.