Path Following Control for an Underactuated USV with Offline Reinforcement Learning Approach

This paper proposes a path following control framework based on offline reinforcement learning (RL) for underactuated unmanned surface vehicle (USV). Addressing challenges of limited actuation and environmental disturbances, the Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning algorithm (TD3+BC) is employed to mitigate extrapolation errors and enhance system stability via behavior cloning regularization. The training data is generated offline using Nonlinear Model Predictive Control (NMPC) to reduce interaction costs. A composite reward function penalizes tracking errors and thrust variations. High-fidelity simulations under hydrodynamic and wind disturbances in Unreal Engine validate the method's robustness, demonstrating stable convergence to a circular reference path. The results highlight offline RL as an effective and stable alternative to online approaches for underactuated USV control.