Application of Reinforcement Learning in Deep-Stall Recovery

The aircraft deep-stall phenomenon is caused by an angle of attack (AOA) that stabilizes at an equilibrium point of high AOA. This excessive AOA leads to a reduction in the lift, as well as a decrease in the elevator efficiency, making it difficult to recover the aircraft out of this very dangerous flight condition. Reinforcement learning (RL) methods offer a design approach for such complex nonlinear control problems. However, during deep-stall recovery tasks, the nonlinearity of the aircraft model is high, and the control efficiency is substantially reduced, hence limiting the application of direct RL methods. To address this problem, we conduct bifurcation and phase plane analyses on the deep-stall model of the aircraft, and use the results as domain knowledge to construct the reward function. Then, we apply the proximal policy optimization algorithm to the deep-stall strategy. Finally, in the simulation, we compare the method with feedback shaping with the RL method without feedback shaping. The simulation results indicate that although the former method recovers the aircraft at the AOA, its uncontrollable state renders this an unsuccessful recovery. Meanwhile, the proposed method stably performs deep-stall recovery tasks through a loop maneuver.