Deep Reinforcement Learning-Based Intelligent Decision-Making for Orbital Game of Satellite Swarm

Recent years have witnessed the rapid development of aerospace science and technology, and the orbital game technology has shown great potential value in the field of failed satellite maintenance, debris removal, etc. In this case, orbital game is often characterized by nonlinear dynamic model, unknown state information, high randomness, but the existing approaches to deal with game problem are difficult to be applied. The analytical method based on game theory is only applicable to simple scenarios, and it is challenging to find the optimal strategy for such complex scenarios as satellite swarm game. It should be noted that deep reinforcement learning has some research basis in the cooperative decision-making and control of multi-agents. In view of its powerful perception and decision ability, this paper applies deep reinforcement learning to solve the orbital game problem of satellite swarm. Firstly, the game scenario is modeled, where typical constraints, e.g., minimum time, optimal fuel, and collision avoidance, are taken into consideration in the game process, and then the multi-agent reinforcement learning algorithm is developed to solve the optimal maneuver strategy. The algorithm is based on the Actor-Critic architecture and uses a centralized training and decentralized execution approach to solve the optimal joint maneuver strategy. For different task scenarios, the action space, state observation space, and reward space are designed to introduce more rewards that match the specific game tasks to make the algorithm converge quickly, so that the satellite swarm emerges and executes better intelligent strategies to complete the corresponding game task.