Integral-Reinforcement-Learning-Based Hierarchical Optimal Evolutionary Strategy for Continuous Action Social Dilemma Games

This article presents a framework for exploring optimal evolutionary strategies in continuous-action social dilemma games with a hierarchical structure comprising a leader and multifollowers. Previous studies in game theory have frequently overlooked the hierarchical structure among individuals, assuming that decisions are made simultaneously. Here, we propose a hierarchical structure for continuous action games that involves a leader and followers to enhance cooperation. The optimal evolutionary strategy for the leader is to guide the followers' actions to maximize overall benefits by exerting minimal control, while the followers aim to maximize their payoff by making minimal changes to their strategies. We establish the coupled Hamilton-Jacobi-Bellman (HJB) equations to find the optimal evolutionary strategy. To address the complexity of asymmetric roles arising from the leader-follower structure, we introduce an integral reinforcement learning (RL) algorithm known as two-level heuristic dynamic programming (HDP)-based value iteration (VI). The implementation of the algorithm utilizes neural networks (NNs) to approximate the value functions. Moreover, the convergence of the proposed algorithm is demonstrated. Additionally, three social dilemma models are presented to validate the efficacy of the proposed algorithm.