LMA: lightweight mixed-domain attention for efficient network design

Attention mechanisms, benefiting from the capability of modeling feature inter-dependencies among channels or spatial locations, have been demonstrated to have great potential in improving the performance of deep convolutional neural networks. However, most existing methods are dedicated to separately developing more intricate channel attention or spatial attention modules to achieve good performance, which inevitably results in losing important information and increasing model overhead. To alleviate this dilemma, in this paper, we propose a novel architecture unit called the lightweight mixed-domain attention (LMA) module. First, LMA aggregates spatial features by using two direction-aware 1D average pooling, which not only captures contextual long-range dependencies but also retains accurate positional information. Subsequently, it adaptively models inter-channel relationships by utilizing our proposed nonlinear local cross-channel interaction strategy, substantially decreasing model overhead while maintaining competitive performance. Our LMA is lightweight yet efficient and can be flexibly plugged into various classic backbones including lightweight MobileNetV2 and heavyweight ResNets as a plug-and-play module. Extensive experimental results of image classification on ImageNet-1K and object detection and instance segmentation on MS COCO demonstrate the superiority of our method against state-of-the-art (SOTA) counterparts. Furthermore, we verify our advanced philosophy through the Grad-CAM++ visualization results.