Federated Multi-Agent Reinforcement Learning with Knowledge Reuse for Robust Cooperation in Edge-Intelligent IoT Systems

The rapid proliferation of intelligent devices in the Internet of Things (IoT) has intensified the demand for scalable and robust cooperative intelligence across distributed agents. In edge-centric multi-robot and multi-UAV systems, challenges such as intermittent wireless communication, dynamic environments, and decentralized data distribution hinder the deployment of conventional multi-agent reinforcement learning (MARL) methods. To address these issues, we propose a federated multi-agent reinforcement learning framework that enables decentralized training with reliable knowledge reuse for enhanced cooperation. Each agent independently maintains actor-critic networks and periodically exchanges partial experiences with a central server to facilitate federated policy aggregation and Q-value refinement. To mitigate policy mismatch and out-of-distribution (OOD) risks caused by communication delays and stale experiences, we introduce a divergence-aware experience filtering mechanism and an adaptively updated central buffer. This synergistic integration of federated actor training and conservative experience sharing enhances learning efficiency and policy robustness under communication uncertainty. Theoretical analysis verifies bounded estimation error and stability. Extensive experiments on benchmark MARL tasks (MPE, SMAC) demonstrate superior performance in terms of episode return, sample efficiency, and resilience to network disconnections. Moreover, simulations in a realistic edge-intelligent IoT system further demonstrate the practical effectiveness, scalability, and communication efficiency of the proposed framework under resource constraints and intermittent connectivity.