Hierarchical dynamic graphical games for optimal leader-follower consensus control

In this paper, a hierarchical dynamic graphical game (HDGG) model is proposed to extend the existing optimal leader-follower consensus control framework for a class of linear stochastic systems. Owing to the interaction relationships and communication constraints among agents determined by the communication topology, the multi-agent system naturally exhibits a multi-stage hierarchical decision-making mechanism that relies solely on the local perceptions available to each agent. The proposed model introduces a multi-layer distributed structure through sequential decision-making, which accounts for both in-degree and out-degree tracking errors in a generalized leader-follower sense. Following the bottom-up principle, hierarchical and simultaneous optimal control policies for each agent are derived from the bottom to the top in a dynamic stochastic environment. A corresponding policy gradient algorithm is developed for the implementation of HDGG under both known and unknown parameter settings, and its convergence and optimality are guaranteed under certain conditions. Furthermore, the control policies are shown to constitute a Stackelberg-Nash equilibrium owing to the unique solution of the corresponding Riccati equation. Finally, an illustrative example is provided to demonstrate the efficacy of the proposed algorithm in both model-based and model-free settings.