Mix-attention approximation for homogeneous large-scale multi-agent reinforcement learning

In large-scale multi-agent environments with homogeneous agents, most works provided approximation methods to simplify the interaction among agents. In this work, we propose a new approximation, termed mix-attention approximation, to enhance multi-agent reinforcement learning. The approximation is made by a mix-attention module, used to form consistent consensuses for agents in partially observable environments. We leverage the hard attention to compress the perception of each agent to some more partial regions. These partial regions can engage the attention of several agents at the same time, and the correlation among these partial regions is generated by a soft-attention module. We give the training method for the mix-attention mechanism and discuss the consistency between the mix-attention module and the policy network. Then we analyze the feasibility of this mix-attention-based approximation, attempting to build integrated models of our method into other approximation methods. In large-scale multi-agent environments, the proposal can be embedded into most reinforcement learning methods, and extensive experiments on multi-agent scenarios demonstrate the effectiveness of the proposed approach.